Image processing device, image processing method, system

Abstract

To provide a technique for acquiring a correction amount that makes it possible to control the brightness on an LED wall, which is an array of LED panels, for making it possible to display information of the same brightness even if the LED wall is imaged from any direction.SOLUTION: An imaging area to be imaged by an imaging apparatus on a display screen which is an array of display panels is identified on the basis of parameters of the imaging apparatus. A correction amount for the luminance value of a pixel is obtained on the basis of an angle from the pixel to the imaging apparatus in the imaging area. The correction amount is corrected on the basis of individual difference information for compensating for individual differences in the luminance of the display panel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a brightness control technique in a display panel.

Background Art

[0002] Conventionally, a technique for correcting the video of a display monitor according to the position of an observer has been known (Patent Document 1 and Patent Document 2). The techniques described in Patent Document 1 and Patent Document 2 acquire the position of the observer by an imaging device such as a camera, and correct the video of the display monitor based on the acquired position information.

[0003] Recently, in the field of video production, a method (virtual production), in which a video is displayed on a display device composed of a plurality of LED panels, such as an LED wall, and the video is imaged with a camera, has begun to spread. In virtual production, the movement and line of sight of the camera are measured in real time, and the video to be displayed on the LED wall portion included in the angle of view of the camera is changed in real time. By shooting this with a camera, a video as if there is a real object there is shot.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] LED elements, which make up the pixels of LED walls commonly used in virtual production, have an angular distortion characteristic. It is known that the brightness of the LED elements decreases when viewed from an oblique angle compared to when viewed directly. Furthermore, while conventional display monitors, which are the target of correction, are display devices composed of a single LED panel, LED walls are composed of multiple LED panels. Therefore, if a single correction coefficient is defined for the entire set of multiple LED panels, there is a problem in that the variation in brightness of each panel cannot be controlled.

[0006] The present invention provides a technique for obtaining a correction amount that enables brightness control in an LED wall, which is an array of LED panels, so that display information with the same brightness can be captured regardless of the direction from which the LED wall is photographed. [Means for solving the problem]

[0007] One aspect of the present invention is a means for identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on the parameters of the imaging device, Based on the angle from the pixel to the imaging device in the imaging area, the amount of correction for the brightness value of the pixel is obtained, and the individual differences in brightness of the display panel are corrected. The correction coefficient and the correction coefficient for correcting the vignetting of the lens are multiplied by the correction amount. Correction means for correcting the correction amount and It is characterized by being equipped with [the following features]. [Effects of the Invention]

[0008] According to the present invention, it is possible to obtain a correction amount that enables brightness control in an LED wall, which is an array of LED panels, so that display information of the same brightness can be captured regardless of the direction from which the LED wall is photographed. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example of the system configuration. [Figure 2]A block diagram showing an example of a computer device hardware configuration applicable to the image processing device 200. [Figure 3] A block diagram showing a more detailed example of the functional configuration of the image processing device 200. [Figure 4] A flowchart of the processes performed by the image processing device 200 to display information on the LED wall. [Figure 5] A diagram illustrating an example of a method for determining the correction region. [Figure 6] A flowchart showing the details of the process in step S505. [Figure 7] A diagram illustrating the process in step S703. [Figure 8] A diagram showing an example of a table structure. [Figure 9] A diagram illustrating an example of individual variation information. [Figure 10] A block diagram showing an example of the system configuration. [Figure 11] A block diagram showing a more detailed example of the functional configuration of the image processing device 200. [Figure 12] A flowchart of the processes performed by the image processing device 200 to display information on the LED wall. [Figure 13] A flowchart showing the details of the process in step S1301. [Figure 14] A diagram illustrating the process in step S704. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] First, a configuration example of the system according to this embodiment will be described using the block diagram of FIG. 1. As shown in FIG. 1, the system according to this embodiment includes a display device 300 that provides a display screen which is an array of display panels (arrays of display elements), an imaging device 100 that images a part or all of the display screen, and an image processing device 200 that obtains a correction amount of the luminance of pixels on the display screen and outputs it to the display device 300 together with display information. Note that between the image processing device 200 and the display device 300, and between the image processing device 200 and the imaging device 100, they are configured to be able to perform data communication with each other via a wired network, a wireless network, a network combining wired and wireless, or the like.

[0012] First, the display device 300 will be described. The display device 300 includes display panels 302 and 303 capable of displaying display information such as images and characters. In this embodiment, by arranging the display panel 302 and the display panel 303 side by side (adjacent to each other), a display screen which is an array of the display panel 302 and the display panel 303 is provided. In this embodiment, the display panel 302 and the display panel 303 are assumed to be LED panels (arrays of LED elements (pixels)). In this case, the display device 300 (the display screen which is an array of the display panel 302 and the display panel 303) becomes an LED wall. Note that in FIG. 1, for simplicity of explanation, the number of display panels included in the display device 300 (that is, the number of display panels constituting the LED wall) is set to 2, but any number of 2 or more is acceptable.

[0013] Further, the display device 300 includes a control unit 301. The control unit 301 performs various operation controls of the display device 300. For example, the control unit 301 divides the display information transmitted from the image processing device 200 into first display information to be displayed on the display panel 302 and second display information to be displayed on the display panel 303. In this embodiment, the display information transmitted from the image processing device 200 is divided into the left half and the right half, with the left half being the first display information and the right half being the second display information. Then, the control unit 301 causes the first display information to be displayed on the display panel 302 and the second display information to be displayed on the display panel 303.

[0014] For example, suppose the entire LED wall can display information for a vertical A pixel and a horizontal B pixel area. Here, the coordinates of the upper left corner of the LED wall are (0,0). In this case, display panel 302 will be responsible for displaying the area where the upper left corner coordinates are (0,0) and the lower right corner coordinates are (B / 2-1,A-1). Display panel 303 will be responsible for displaying the area where the upper left corner coordinates are (B / 2,0) and the lower right corner coordinates are (B-1,A-1). In this way, once the arrangement of the display panels is determined, it is determined which display panel is responsible for displaying which area of ​​the information that can be displayed on the entire LED wall.

[0015] The control unit 301 displays the display information transmitted from the image processing device 200 by illuminating LED elements (pixels) at specified positions on the display board 302 or the display board 303.

[0016] Next, the imaging device 100 will be described. The imaging device 100 is a camera that captures part or all of the LED wall. The imaging device 100 may be a camera that captures moving images, or a camera that captures still images periodically or irregularly. The imaging device 100 may also be a device equipped with such a camera (such as a tablet terminal device with a camera or a smartphone).

[0017] The imaging unit 101 includes a lens, a drive control unit that drives and controls the lens, an image sensor that converts light entering from the outside world through the lens into an image signal through photoelectric conversion, and an image processing circuit that generates an image by performing various image processing based on the image signal.

[0018] The gyro sensor 102 measures its own position and orientation as the position and orientation of the imaging device 100, respectively, and outputs position information indicating the position of the imaging device 100 and orientation information indicating the orientation of the imaging device 100.

[0019] The transmitting / receiving unit 103 performs data communication with the image processing device 200. The transmitting / receiving unit 103 transmits to the image processing device 200 as shooting information the position information and orientation information output from the gyro sensor 102, the captured image generated by the imaging unit 101, and the "focal length information indicating the focal length of the imaging device 100" and "sensor size information indicating the vertical and horizontal dimensions of the image sensor" possessed by the imaging unit 101.

[0020] Furthermore, the imaging device 100 may also have a recording unit that records, for example, the captured image generated by the imaging unit 101 and information related to the captured image in its own memory or on a memory card that is detachably attached to the imaging device 100.

[0021] Next, the image processing device 200 will be described. The image processing device 200 is a computer device such as a PC (personal computer), tablet terminal, or smartphone, and performs various processes through communication with the imaging device 100 and the display device 300. The image processing device 200 identifies the imaging area to be imaged by the imaging device 100 on the LED wall based on the parameters of the imaging device 100. The image processing device 200 then obtains the brightness correction amount of the LED element (pixel) based on the angle from the LED element (pixel) to the imaging device 100 in the imaging area, and corrects the correction amount based on information for correcting individual differences in brightness of the display panel. The image processing device 200 then outputs the corrected correction amount along with the display information to the display device 300. The display device 300 makes the LED element (pixel) emit light at a brightness value corrected according to the brightness value of the LED element (pixel) according to the brightness value of the LED element (pixel) corresponding to the display information output from the image processing device 200. A more detailed example of the functional configuration of the image processing device 200 is shown in the block diagram of Figure 3.

[0022] Next, the process performed by the image processing device 200 to display information on the LED wall will be explained according to the flowchart in Figure 4. In step S501, the calculation unit 202 receives (acquires) the imaging information transmitted from the imaging device 100. More specifically, the first input unit 220 of the calculation unit 202 acquires the position information and orientation information transmitted from the imaging device 100, and the second input unit 221 of the calculation unit 202 acquires the focal length information and sensor size information transmitted from the imaging device 100.

[0023] In step S503, the first calculation unit 222 of the calculation unit 202 acquires the placement information stored in the holding unit 201. The placement information is information indicating the "positions of display board 302 and display board 303" in the same coordinate system (hereinafter referred to as the shared coordinate system) as the position and orientation measured by the gyro sensor 102. The positions of the LED elements on display board 302 and display board 303 are known. Therefore, the positions of the "LED elements on display board 302 and display board 303" in the shared coordinate system can be obtained from the placement information and the positions of the LED elements on display board 302 and display board 303. The placement information may also be information indicating the positions of the "LED elements on display board 302 and display board 303" in the shared coordinate system. Furthermore, the "normal vector of the LED wall" in the shared coordinate system can be obtained from the placement information and the positions of the LED elements on display board 302 and display board 303. The placement information may also include the "normal vector of the LED wall" in the shared coordinate system.

[0024] In the following explanation, we will assume that the positions of the LED elements on display boards 302 and 303, respectively, and the normal vector of the LED wall in the shared coordinate system are known based on the placement information.

[0025] In step S504, the first calculation unit 222 receives the imaging information acquired in step S501, and in step S50 3Using the acquired placement information, the area within the field of view of the imaging device 100 on the LED wall is determined as the correction area. In other words, the correction area is the imaging area captured by the imaging device 100 on the LED wall.

[0026] Various methods can be applied to determine the correction region, and in this embodiment, any method may be used to determine the correction region. An example of a method for determining the correction region is described below with reference to Figure 5.

[0027] First, the first calculation unit 222 determines the horizontal and vertical field of view of the imaging device 100 by calculations known from the focal length information and sensor size information. Next, the first calculation unit 222 determines a vector 603 from the position 602 of the imaging device 100 indicated by the position information to the line of sight direction indicated by the attitude information. Vector 607 is a vector indicating the direction directly upward in a local coordinate system based on the attitude of the imaging device 100 indicated by the attitude information. Then, the first calculation unit 222 determines a vector 604A from position 602 that points in the direction of "one end of the horizontal field of view of the imaging device 100 in a local coordinate system where vector 603 is the line of sight direction and vector 607 is the direction directly upward." The first calculation unit 222 also determines a vector 604B from position 602 that points in the direction of "the other end of the horizontal field of view of the imaging device 100 in a local coordinate system where vector 603 is the line of sight direction and vector 607 is the direction directly upward." The first calculation unit 222 also determines vector 605A from position 602, pointing in the direction of "one end of the vertical field of view of the imaging device 100 in a local coordinate system where vector 603 is the line of sight direction and vector 607 is the upward direction." The first calculation unit 222 also determines vector 605B from position 602, pointing in the direction of "the other end of the vertical field of view of the imaging device 100 in a local coordinate system where vector 603 is the line of sight direction and vector 607 is the upward direction."

[0028] Furthermore, the first calculation unit 222 obtains the vertical and horizontal directions of the LED wall 500 in the shared coordinate system from the positions of the LED elements in the display board 302 and the display board 303 in the shared coordinate system, and then determines the angle θA formed by the vertical direction and vector 607. This angle θA represents the roll angle of the imaging device 100 with respect to the line of sight. The first calculation unit 222 determines the intersection position U with vector 605A, the intersection position D with vector 605B, the intersection position R with vector 604A, and the intersection position L with vector 604B in the LED wall 500. The first calculation unit 222 then identifies a rectangle (shown as a dotted line in Figure 5) enclosed by a line obtained by rotating a horizontal line passing through intersection point U by θA, a line obtained by rotating a horizontal line passing through intersection point D by θA, a line obtained by rotating a vertical line passing through intersection point R by θA, and a line obtained by rotating a vertical line passing through intersection point L by θA as the correction region (the rectangle shown as a dotted line in Figure 5). In addition, to accommodate sudden movements of the imaging device 100, the correction region may be defined as the region encompassing the region 606 identified by the above method.

[0029] In step S505, the second calculation unit 223 of the calculation unit 202 determines the correction amount for correcting the brightness value of each LED element (pixel) in the correction region determined in step S504. The LED element has R (red), G (green), and B (blue) elements, and in step S505, the second calculation unit 223 determines the correction amount for correcting the brightness value of the R element, the correction amount for correcting the brightness value of the G element, and the correction amount for correcting the brightness value of the B element. Details of the processing in step S505 will be described later.

[0030] In step S506, the generation unit 203 in the calculation unit 202 generates display information to be displayed on the LED wall based on the image and character data stored in the holding unit 201. The display information includes information to be displayed in the correction area and information to be displayed outside the correction area. In step S507, the generation unit 203 outputs the display information generated in step S506 and the correction amount obtained in step S505 to the display device 300.

[0031] Next, the details of the process in step S505 described above will be explained according to the flowchart in Figure 6. In step S703, the second calculation unit 223 determines the angle from the position of each LED element (pixel) in the correction region toward the position of the imaging device 100 indicated by the position information. The process in step S703 will be explained using Figure 7. In Figure 7, region 800 is a part of the LED wall and includes region 606, which is the correction region. Each rectangle in region 800 represents an LED element (pixel), and the coordinates of rectangle 801 at the upper left corner of region 800 are (x, y). At this time, the coordinates of the rectangle at the upper left corner of region 606 are (x+3, y+1), and the coordinates of the rectangle at the lower right corner are (x+5, y+3).

[0032] Here, when determining the angle corresponding to an LED element (pixel) at coordinate (x+3, y+1), the second calculation unit 223 determines the angle between the normal vector 803 of the LED wall and the vector 802 pointing from coordinate (x+3, y+1) to position 602 as the angle corresponding to the LED element (pixel) at coordinate (x+3, y+1). The second calculation unit 223 can determine the angle corresponding to each LED element (pixel) by performing this process for each LED element (pixel) included in the region 606.

[0033] The angle corresponding to an LED element (pixel) at coordinate (x+3, y+1) is expressed in two components. In the following, as shown in Figure 7, the horizontal direction of the LED wall is the x-axis direction, the vertical direction of the LED wall is the y-axis direction, and the direction of the normal vector 803 of the LED wall (each LED element) is the z-axis direction. Here, the normal vector 803 (unit vector) = (a1, a2, a3) and the vector 802 (unit vector) = (b1, b2, b3). At this time, the second calculation unit 223 can find the angle (θXZ, θXY) corresponding to the LED element (pixel) at coordinate (x+3, y+1) by calculating the following equations (1) and (2).

[0034]

number

[0035] In step S704, the second calculation unit 223 acquires the "luminance value correction amount" corresponding to the angle obtained in step S703 for each pixel in the correction region. Here, the holding unit 201 has a table, as illustrated in Figure 8, pre-registered.

[0036] The table in Figure 8 registers the correction amounts for the brightness values ​​of LED elements (pixels) corresponding to angles of -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees. These correction amounts are for the R (red) brightness value (R element correction amount), the G (green) brightness value (G element correction amount), and the B (blue) brightness value (B element correction amount). This table takes the angle of the LED element to the imaging device 100 directly facing it as 0 degrees, and represents the angle from the left or right of the LED element, and the degree of the angle at which the imaging device 100 is imaging the LED element as a signed angle, showing the correction amount corresponding to that angle. In Figure 8, the angle from the LED element to the left side of the imaging device 100 is represented as a negative angle, and the angle from the LED element to the right side of the imaging device 100 is represented as a positive angle.

[0037] Here, Figure 8 shows the correction amount corresponding to the angle formed with the Z-axis (normal vector 803) in the XZ plane. However, in reality, as shown in Figure 14, a correction amount is required corresponding to the angle formed between the vector from the LED element position 1599 to various positions on the hemisphere centered on the center 1599 and the normal vector 803.

[0038] In this hemisphere, point 1503 is the point on the imaging device 100 that is directly opposite the LED element at position 1599, i.e., the point where the angle is 0 degrees. In this hemisphere, the left half is represented by negative angles and the right half by positive angles, with the normal vector 803 as the center. For example, in the hemisphere, the point on the arc where Y=0 (referred to as the basic arc) with an angle of "-30 degrees" is point 1504, as it is 30 degrees to the left of the normal vector 803. The table in Figure 8 is a table in which correction amounts corresponding to points on the basic arc where the angle with the normal vector 803 is -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees are registered.

[0039] To prepare correction amounts corresponding to the angles formed by the vectors from position 1599 to various positions on the hemisphere and the normal vector 803, the correction amounts corresponding to "points on the rotated basic arc where the angle with the normal vector 803 is -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees" are calculated by interpolation when the basic arc is rotated around the Z axis by Δ degrees, 2Δ degrees, ..., 180 degrees. For example, the correction amount corresponding to point 1506 (the point on the hemisphere where the angle with the normal vector 803 is "-80 degrees" in the arc where X=0) is the average of the correction amount CA1 corresponding to the point where the angle with the normal vector 803 is "-80 degrees" in the basic arc and the correction amount CA2 corresponding to the point where the angle with the normal vector 803 is "80 degrees" in the basic arc.

[0040] The correction amount CD corresponding to the point on the rotated basic arc, obtained by rotating the basic arc by S degrees around the Z axis, where the angle with the normal vector 803 is -80 degrees, can be calculated, for example, using the following formula.

[0041] CD = (180 - S) x CA / 180 + S x CB / 180 By performing these calculations for S = Δ degrees, 2Δ degrees, ..., 180 degrees, we can obtain correction amounts corresponding to "points where the angle with the normal vector 803 is -80 degrees" for each of Δ degrees, 2Δ degrees, ..., 180 degrees. Then, we perform these calculations for "points where the angle with the normal vector 803 is θ degrees" (θ = -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, 80 degrees).

[0042] Through this process, the second calculation unit 223 obtains, as reference information, a correction amount corresponding to the angle (θXZ, θXY) formed by the vectors from the LED element position 1599 to various positions on the hemisphere and the normal vector 803.

[0043] In this embodiment, a table was used that registered correction amounts corresponding to angles of -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees. However, the number of angles registered is not limited to a specific number. For example, a table with correction amounts for a larger number of angles may be used.

[0044] Furthermore, in this embodiment, the correction amount for various angles was determined from the table in Figure 8. However, such processing may be performed in advance to determine the correction amount for various angles, and this data may be registered in the holding unit 201.

[0045] The second calculation unit 223 then refers to the reference information and obtains a correction amount for the brightness value corresponding to the angle obtained in step S703 for each LED element (pixel) in the correction region. Note that when the reference information is generated using the table in Figure 8, no correction amount is obtained for angles of 80 degrees or more. Therefore, for LED elements (pixels) where the angle obtained in step S703 is 80 degrees or more, the correction amount corresponding to 80 degrees is obtained. Also, if the angle obtained in step S703 is not registered in the reference information, an interpolated correction amount is obtained by interpolating from the correction amount corresponding to an angle close to the angle obtained in step S703 among the angles registered in the reference information, and this is obtained as the correction amount corresponding to the angle obtained in step S703.

[0046] Furthermore, although this embodiment describes the LED wall as having LED elements with R, G, and B components, it may also have LED elements of other colors, such as white LEDs (W). In this case, it is necessary to obtain correction amounts corresponding to the other color LED elements in addition to the correction amounts for R, G, and B.

[0047] Next, in step S705, the second calculation unit 223 performs a correction process on the correction amount obtained in step S704 to reduce the influence caused by individual differences in the brightness of the display panels 302 and 303. LED panels may produce different brightness output images even when the same video signal is applied, due to variations in the precision of the components during manufacturing. Therefore, in this embodiment, correction coefficients for R, G, and B are registered in the holding unit 201 as individual difference information to correct for individual differences in the brightness of the LED elements in each of the display panels 302 and 303. An example of individual difference information is shown in Figure 9.

[0048] As shown in Figure 9, the individual difference information includes, for each of the display panels 302 and 303, the position of the display panel in the shared coordinate system (panel position), the correction amount for R (correction amount R), the correction amount for G (correction amount G), and the correction amount for B (correction amount B).

[0049] The second calculation unit 223 corrects the LED elements (pixels) belonging to the display panel 302 within the correction area by multiplying the correction amounts R, G, and B obtained in step S704 by the correction amounts R, G, and B corresponding to the positions of the display panel 302 indicated by the placement information. Similarly, the second calculation unit 223 corrects the LED elements (pixels) belonging to the display panel 303 within the correction area by multiplying the correction amounts R, G, and B obtained in step S704 by the correction amounts R, G, and B corresponding to the positions of the display panel 303 indicated by the placement information.

[0050] Then, in step S706, the second calculation unit 223 outputs the "correction amount for each LED element (pixel) in the correction region" corrected in step S705 to the generation unit 203.

[0051] Next, the operation of the display device 300, which has acquired display information and correction amounts from the image processing device 200, will be described. The control unit 301 determines the brightness value of each pixel as the brightness value of the pixel obtained from the image processing device 200 by multiplying the brightness value of each pixel in the correction area of ​​the display information acquired from the image processing device 200 by the correction amount acquired for that pixel.

[0052] For example, suppose that pixel P at coordinates (a,b) within the correction area is a pixel on the display panel 302, the brightness value of pixel P is (R,G,B)=(100,100,150), and the correction amounts obtained for pixel P in step S704 are (1.12, 1.11, 1.19). In this case, according to the individual difference information in Figure 9, the panel position of the display panel 302 is (1,1), so the correction amounts R, G, and B corresponding to the panel position (1,1) are 1.01, 1.02, and 1.01, respectively.

[0053] In this case, the control unit 301 obtains from the image processing device 200 a correction amount "1.1312" obtained by multiplying the correction amount "1.12" acquired in step S704 for pixel P by the correction amount R "1.01", and using this correction amount for pixel P.

[0054] Similarly, the control unit 301 sets the correction amount of G obtained in step S704 to "1.11" for pixel P. G The correction amount "1.1322", obtained by multiplying by "1.02", is acquired from the image processing device 200 as the correction amount for G of pixel P.

[0055] Similarly, the control unit 301 sets the correction amount of B obtained in step S704 to "1.19" for pixel P. B The correction amount "1.2019", obtained by multiplying by "1.01", is acquired from the image processing device 200 as the correction amount for B of pixel P.

[0056] The control unit 301 then calculates the corrected R luminance value of pixel P as "113" (rounded to the first decimal place) by multiplying the R luminance value of pixel P, "100", by the R correction amount of pixel P, "1.1312".

[0057] Similarly, the control unit 301 calculates the corrected luminance value of G for pixel P as "113" (rounded to the first decimal place), which is obtained by multiplying the luminance value of G for pixel P, "100", by the correction amount for G of pixel P, "1.1322".

[0058] Similarly, the control unit 301 calculates the corrected brightness value of pixel P's B by multiplying the brightness value of pixel P's B, "150," by the correction amount of pixel P's B, "1.2019," resulting in "180" (rounded to the first decimal place).

[0059] The control unit 301 then controls the light emission of the LED elements such that the R element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "113", the G element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "113", and the B element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "180".

[0060] Thus, in this embodiment, regardless of the relative positional relationship between the imaging device and the LED wall, or the variation in brightness of each LED panel on the LED wall, an image of the LED wall with a constant brightness can be obtained within the field of view of the imaging device.

[0061] In this embodiment, the image processing device 200 outputs display information and a correction amount to the display device 300, and the display device 300 illuminates the LED elements based on the brightness value obtained by correcting the brightness value of the display information according to the correction amount. However, the image processing device 200 may also perform the brightness value correction described above using the display information and the correction amount, and output the corrected brightness value to the display device 300. In this case, the display device 300 controls the illumination of the LED elements according to the brightness value obtained from the image processing device 200.

[0062] Furthermore, in this embodiment, a case was described in which a correction amount corresponding to the angle of each LED element (pixel) is obtained using reference information, and this correction amount is output as display information to the display device 300.

[0063] However, this reference information may be stored in a memory device such as HDD405 or external storage device 409, or it may be transmitted to an external device via network I / F412. With this configuration, when obtaining the correction amount according to the angle of each LED element (pixel) in the future, the reference information stored in the memory device or external device can be retrieved and used. Alternatively, reference information generated by an external device can be retrieved, stored in HDD405, and read and used as needed.

[0064] [Second Embodiment] The following describes the differences from the first embodiment, and unless otherwise specified, it will be assumed that it is the same as the first embodiment. An example of the system configuration according to this embodiment is shown in the block diagram of Figure 10. In the configuration shown in Figure 10, the image processing device 200 has a holding unit 1101 in the system shown in Figure 1.

[0065] A more detailed example of the functional configuration of the image processing device 200 is shown in the block diagram of Figure 11. The configuration shown in Figure 11 is the same as the configuration shown in Figure 3, but with the addition of a holding unit 1101 and a third calculation unit 1222 instead of the first calculation unit 222. The third calculation unit 1222 performs the processing described above, which is the processing performed by the first calculation unit 222, and also acquires peripheral light falloff information in the correction region.

[0066] Next, the processing performed by the image processing device 200 to display information on the LED wall will be explained according to the flowchart in Figure 12. In Figure 12, the same processing steps as those shown in Figure 4 are given the same step numbers, and the explanation of these processing steps will be omitted. In this embodiment, the processing of steps S503 and S504 is performed by the third calculation unit 1222.

[0067] In step S1301, the third calculation unit 1222 acquires "peripheral light falloff information indicating peripheral light falloff for the lens" stored in the holding unit 1211. Peripheral light falloff information is information for correcting peripheral light falloff corresponding to the lens mounted on the imaging device 100, and is, for example, a map that holds "correction coefficients for correcting peripheral light falloff" for each pixel of the image captured by the imaging device 100.

[0068] Here, we will explain peripheral light falloff information. Peripheral light falloff correction information is a map of correction coefficients used to correct peripheral light falloff, which is the decrease in brightness of an image captured by the imaging device 100 as it moves away from the center of the field of view.

[0069] The third calculation unit 1222 then maps the peripheral light falloff information to the correction region and obtains a correction coefficient corresponding to each pixel in the correction region. This "correction coefficient corresponding to each pixel in the correction region" becomes the "peripheral light falloff information of the correction region".

[0070] When mapping peripheral light falloff information to the correction region, there may be areas in the mapped peripheral light falloff information where no correction coefficient exists. In such areas, a correction coefficient should be set by interpolating using the surrounding correction coefficients with a well-known interpolation technique. Note that there is no specific method for obtaining the correction coefficient corresponding to each pixel in the correction region.

[0071] Furthermore, the peripheral light falloff information is not limited to a map that holds a correction coefficient for each pixel of the image captured by the imaging device 100, but may also be a calculation formula that can be applied to correct the peripheral light falloff of pixels on the captured image.

[0072] Furthermore, the holding unit 1101 may also store peripheral light falloff information for each type of lens. In this case, the third calculation unit 1222 acquires the "identification information of the lens attached to the imaging device 100" managed by the imaging unit 101, and acquires peripheral light falloff information corresponding to the identification information.

[0073] The second calculation unit 223 then calculates a correction amount by correcting the correction amount for each pixel in the correction region based on individual difference information, in the same manner as in the first embodiment. The second calculation unit 223 then calculates a correction amount by correcting the corrected correction amount based on the correction coefficient acquired by the third calculation unit 1222.

[0074] The details of the process in step S1301 will be explained according to the flowchart in Figure 13. In Figure 13, the same step numbers are used for processing steps that are the same as those shown in Figure 6, and the explanation for these processing steps will be omitted.

[0075] In step S1401, the third calculation unit 1222 obtains the correction coefficient corresponding to each pixel in the correction region. The second calculation unit 223 then multiplies the correction amount for each pixel corrected in step S705 by the correction coefficient corresponding to that pixel from the correction coefficients in the correction region to obtain the corrected amount. In step S706, the second calculation unit 223 outputs the "correction amount for each pixel in the correction region" corrected in step S1401 to the generation unit 203.

[0076] Thus, in this embodiment, regardless of the relative positional relationship between the imaging device and the LED wall, or the variation in brightness of each LED panel in the LED wall, an image of the LED wall can be obtained with a constant brightness within the field of view of the imaging device, and with the peripheral light falloff of the lens corrected.

[0077] [Third Embodiment] In the first and second embodiments, individual difference information was assumed to be created in advance and registered in the holding unit 201. However, before using this system, individual differences in the brightness of the LED elements in each of the display boards 302 and 303 may be measured, and individual difference information may be generated based on the results of the measurement and registered in the holding unit 201.

[0078] For example, the image processing device 200 may be provided with a second generation unit (not shown), which generates individual difference information using the captured image of the "image displayed on the display board 302" captured by the imaging device 100 and the captured image of the "image displayed on the display board 303" captured by the imaging device 100. More specifically, the same test chart is displayed on both the display board 302 and the display board 303. Examples of test charts include monochrome charts using signal values ​​that make the LED panel a uniform color, such as gray, red, green, and blue. The imaging device 100 then images the display board 302 in a position and orientation that is directly facing the center of the screen of the display board 302 on which such a test chart is displayed, and acquires an image of the display board 302. Similarly, the imaging device 100 images the display board 303 in a position and orientation that is directly facing the center of the screen of the display board 303 on which such a test chart is displayed, and acquires an image of the display board 303.

[0079] The second generation unit obtains the average value R1 of the R pixel values ​​in the image area of ​​the "test chart displayed on display board 302" from the image captured by display board 302. The second generation unit also obtains the average value R2 of the R pixel values ​​in the image area of ​​the "test chart displayed on display board 303" from the image captured by display board 303. Then, using R1 and R2, the second generation unit calculates the correction amount R for display board 302 and the correction amount R for display board 303 and registers them in the individual difference information. For example, the second generation unit registers R2 / R1 as the correction amount R for display board 302 and R1 / R2 as the correction amount R for display board 303 in the individual difference information.

[0080] Furthermore, the second generation unit obtains the average value G1 of the pixel values ​​of G in the image area of ​​the "test chart displayed on the display panel 302" from the captured image of the display panel 302. The second generation unit also obtains the average value G2 of the pixel values ​​of G in the image area of ​​the "test chart displayed on the display panel 303" from the captured image of the display panel 303. Then, using G1 and G2, the second generation unit calculates the correction amount G for the display panel 302 and the correction amount G for the display panel 303 and registers them in the individual difference information. For example, the second generation unit registers G2 / G1 as the correction amount G for the display panel 302 and G1 / G2 as the correction amount G for the display panel 303 in the individual difference information.

[0081] Furthermore, the second generation unit obtains the average value B1 of the pixel values ​​of B in the image area of ​​the "test chart displayed on display board 302" from the captured image of display board 302. The second generation unit also obtains the average value B2 of the pixel values ​​of B in the image area of ​​the "test chart displayed on display board 303" from the captured image of display board 303. Then, using B1 and B2, the second generation unit calculates the correction amount B for display board 302 and the correction amount B for display board 303 and registers them in the individual difference information. For example, the second generation unit registers B2 / B1 as the correction amount B for display board 302 and B1 / B2 as the correction amount B for display board 303 in the individual difference information.

[0082] [Fourth Embodiment] Each functional part of the image processing device 200 shown in Figures 1, 3, 10, and 11 may be implemented in hardware, or each functional part except for the holding units 201 and 1101 may be implemented in software (computer program). In the latter case, a computer device capable of executing such a computer program is applicable to the image processing device 200. An example of a hardware configuration of a computer device applicable to the image processing device 200 will be explained using the block diagram in Figure 2. Such computer devices can include PCs (personal computers), WS (workstations), smartphones, and tablet devices.

[0083] The CPU 401 executes various processes using computer programs and data stored in the RAM 402 and ROM 403. In doing so, the CPU 401 controls the operation of the entire computer system and also executes or controls the various processes described as being performed by the image processing device 200.

[0084] RAM 402 has areas for storing computer programs and data loaded from ROM 403, HDD (hard disk drive) 405, and external storage device 409. RAM 402 also has areas for storing various types of information acquired from the imaging device 100 via the network interface 412. Furthermore, RAM 402 has a work area used by the CPU 401 when executing various processes. In this way, RAM 402 provides various areas as appropriate.

[0085] In addition to the CPU 401 and RAM 402, a GPU may be provided for rendering the display information to be shown on the LED wall (for example, rendering computer graphics), and VRAM may be provided for storing image and text data (computer graphics data, scene data, etc.).

[0086] ROM403 stores configuration data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and so on.

[0087] The auxiliary storage interface 404 is an interface for connecting the HDD 405 to the system bus 408. The HDD 405 stores the OS (operating system), computer programs and data for executing or controlling the various processes described above as processes performed by the image processing device 200 on the CPU 401, and so on. The computer programs and data stored on the HDD 405 are loaded into the RAM 402 as appropriate according to the control of the CPU 401, and become the target of processing by the CPU 401. In addition to or instead of the HDD 405, various storage devices such as optical disc drives and flash memory can also be used.

[0088] Input I / F 406 is a serial bus interface such as USB or IEEE1394, and is an interface for connecting the input device 411 and the external storage device 409 to the image processing device 200.

[0089] The input device 411 is a user interface such as a keyboard, mouse, or touch panel, and can input various instructions to the CPU 401 by being operated by the user.

[0090] The external storage device 409 is a storage device that can be attached to or removed from the image processing device 200, and is a storage device such as a hard disk drive, memory card, CF card, SD card, or USB memory. Some or all of the computer programs and data described as being stored on the HDD 405 may be stored on the external storage device 409.

[0091] The above-mentioned holding units 201 and 1101 can be implemented using, for example, one or more memory devices from among RAM 402, ROM 403, HDD 405, and external storage device 409, as shown in Figure 2.

[0092] Output I / F407, like input I / F406, is a serial bus interface such as USB or IEEE1394, and is an interface for connecting the monitor 410 and display device 300 to the image processing device 200. Output I / F407 may also be a video output terminal such as DVI or HDMI®.

[0093] The monitor 410 is a display device having an LCD screen or a touch panel screen, and displays the processing results of the CPU 401 in the form of images, text, etc. For example, the CPU 401 displays the data it has processed (for example, the real-time arrangement status of the imaging device 100 and the display device 300) on the monitor 410.

[0094] Network I / F412 is a connector for connecting to a network such as Ethernet, and is an interface for connecting the imaging device 100 to the image processing device 200.

[0095] The CPU 401, RAM 402, ROM 403, auxiliary storage I / F 404, input I / F 406, output I / F 407, and network I / F 412 are all connected to the system bus 408. Note that the configuration shown in Figure 2 is an example of a computer device hardware configuration applicable to the image processing device 200, and can be modified or altered as appropriate.

[0096] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) acquisition method / destination / source / storage location, etc., used in each of the above embodiments are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0097] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0098] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0099] The inventions described herein include the following image processing apparatus, image processing method, and computer program.

[0100] (Item 1) A means for identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on the parameters of the imaging device, Based on the angle from the pixel to the imaging device in the imaging area, a correction amount for the brightness value of the pixel is obtained, and a correction means corrects the correction amount based on individual difference information for correcting individual differences in brightness of the display panel. An image processing apparatus characterized by comprising:

[0101] (Item 2) The image processing apparatus according to item 1, characterized in that the identifying means identifies an area included in the field of view range of the imaging device on the display screen as the imaging area, based on the parameters of the imaging device.

[0102] (Item 3) The correction means is The angle between the vector from the pixel to the imaging device and the normal vector of the display screen is determined, and the correction amount corresponding to this angle is obtained as the correction amount for the brightness of the pixel. The correction amount for the brightness of the aforementioned pixels is multiplied by a correction coefficient for correcting individual differences in brightness of the display panel, thereby correcting the correction amount. An image processing apparatus according to item 1 or 2, characterized by the features described herein.

[0103] (Item 4) The correction means corrects the correction amount by multiplying it by a correction coefficient for correcting individual differences in the brightness of the display panel and a correction coefficient for correcting peripheral light falloff of the lens. An image processing apparatus according to any one of items 1 to 3, characterized by the above.

[0104] (Item 5) moreover, An image processing apparatus according to any one of items 1 to 4, characterized by comprising an output means for outputting display information to be displayed on the display screen and a correction amount corrected by the correction means to a display device providing the display screen.

[0105] (Item 6) moreover, An image processing apparatus according to any one of items 1 to 5, characterized by comprising a generation means for generating individual difference information based on the pixel values ​​of the image region of the test chart in the captured image of the test chart displayed on each display panel.

[0106] (Item 7) The image processing apparatus according to any one of items 1 to 6, characterized in that the display panel is an LED panel and the pixels are LED elements in the LED panel.

[0107] (Item 8) Imaging device and Image processing device and A display device that provides a display screen which is an array of display panels and A system having, The aforementioned image processing device is The display screen includes a means for identifying the imaging area captured by the imaging device based on the parameters of the imaging device, A correction means that obtains a correction amount for the brightness value of a pixel based on the angle from the pixel to the imaging device in the imaging area, and corrects the correction amount based on information for correcting individual differences in brightness of the display panel, An output means that outputs the display information and the correction amount corrected by the correction means to the display device. Equipped with, The aforementioned display device is The system includes a control means for correcting the brightness value of each pixel in the imaging area of ​​the display information according to the correction amount of the pixel and displaying it on the display screen. A system characterized by the following features.

[0108] (Item 9) An image processing method performed by an image processing device, The identification means of the image processing device includes an identification step of identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on the parameters of the imaging device, The correction means of the image processing apparatus includes a correction step of obtaining a correction amount for the brightness value of a pixel based on the angle from the pixel to the imaging device in the imaging area, and correcting the correction amount based on individual difference information for correcting individual differences in brightness of the display panel. An image processing method characterized by comprising:

[0109] (Item 10) A computer program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 7.

[0110] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0111] 100: Imaging device 101: Imaging unit 102: Gyro sensor 103: Transceiver / receiver device 200: Image processing device 201: Holding unit 202: Calculation unit 203: Generation unit 300: Display device 301: Control unit 302: Display panel 303: Display panel

Claims

1. A means for identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on the parameters of the imaging device, Based on the angle from the pixel to the imaging device in the imaging area, a correction amount for the brightness value of the pixel is obtained, and a correction means is used to correct the correction amount by multiplying the correction amount by a correction coefficient for correcting individual differences in brightness of the display panel and a correction coefficient for correcting peripheral light falloff of the lens. An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the identifying means identifies an area included in the field of view range of the imaging device on the display screen as the imaging area, based on the parameters of the imaging device.

3. moreover, The image processing apparatus according to claim 1, further comprising an output means for outputting display information to be displayed on the display screen and a correction amount corrected by the correction means to a display device providing the display screen.

4. moreover, The image processing apparatus according to claim 1, further comprising a generation means for generating a correction coefficient for correcting individual differences in brightness of the display panel based on the pixel values ​​of the image region of the test chart in the captured image of the test chart displayed on each display panel.

5. The image processing apparatus according to claim 1, characterized in that the display panel is an LED panel, and the pixels are LED elements in the LED panel.

6. Imaging device and Image processing device and A display device that provides a display screen which is an array of display panels and A system having, The aforementioned image processing device is The display screen includes a means for identifying the imaging area captured by the imaging device based on the parameters of the imaging device, Based on the angle from the pixel to the imaging device in the imaging area, a correction amount for the brightness value of the pixel is obtained, and a correction means for correcting the correction amount by multiplying the correction amount by a correction coefficient for correcting individual differences in brightness of the display panel and a correction coefficient for correcting peripheral light falloff of the lens, An output means that outputs the display information and the correction amount corrected by the correction means to the display device. Equipped with, The aforementioned display device is The system includes a control means for correcting the brightness value of each pixel in the imaging area of ​​the display information according to the correction amount of the pixel and displaying it on the display screen. A system characterized by the following features.

7. An image processing method performed by an image processing device, The identification means of the image processing device includes an identification step of identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on the parameters of the imaging device, The correction means of the image processing apparatus obtains a correction amount for the brightness value of a pixel based on the angle from the pixel to the imaging device in the imaging area, and performs a correction step of correcting the correction amount by multiplying it by a correction coefficient for correcting individual differences in brightness of the display panel and a correction coefficient for correcting peripheral light falloff of the lens. An image processing method characterized by comprising:

8. A computer program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 5.

Images