Image processing device, image processing method, and system

Abstract

To provide a technique for acquiring a correction amount that makes it possible to control the brightness on an LED wall, which is an array of LED panels, for making it possible to display information of the same brightness even if the LED wall is imaged from any direction.SOLUTION: An imaging area to be imaged by an imaging apparatus on a display screen which is an array of display panels is identified on the basis of parameters of the imaging apparatus. A correction amount for the luminance value of a pixel is obtained on the basis of an angle from the pixel to the imaging apparatus in the imaging area. The correction amount is corrected on the basis of individual difference information for compensating for individual differences in the luminance of the display panel.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a brightness control technique for a display panel. [Background technology]

[0002] Conventionally, there have been known techniques for correcting an image on a display monitor according to the position of an observer (Patent Documents 1 and 2). The techniques described in Patent Documents 1 and 2 acquire the position of an observer by an imaging device such as a camera, and correct the image on the display monitor based on the acquired position information.

[0003] Recently, in the field of video production, a technique called virtual production has become popular in which images are displayed on a display device made up of multiple LED panels, such as LED walls, and the images are then captured by a camera. In virtual production, the camera's movement and line of sight are measured in real time, and the images displayed on the LED wall within the camera's field of view are changed in real time. By capturing this with a camera, images are shot that make it seem as if the real thing was actually there. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-42804 A [Patent Document 2] JP 2009-128381 A Summary of the Invention [Problem to be solved by the invention]

[0005] The display elements (LED elements) that make up the pixels of LED walls, which are often used in virtual production, have a distortion characteristic, and it is known that the brightness of the LED elements decreases when observed from an oblique direction compared to when observed directly in front of them. Furthermore, the display monitor that is the subject of correction in conventional technology is a display device composed of a single LED panel, whereas an LED wall is composed of multiple LED panels. Therefore, if one correction coefficient is defined for the entire multiple LED panels, there is an issue that it is not possible to control the brightness variation of each panel.

[0006] The present invention provides a technology for acquiring a correction amount that enables brightness control of an LED wall, which is an array of LED panels, so that display information can be captured with similar brightness regardless of the direction from which the LED wall is captured. [Means for solving the problem]

[0007] One aspect of the present invention is characterized in that it comprises an identification means for identifying an imaging area captured by an imaging device on a display screen which is an array of display panels, based on parameters of the imaging device, and a correction means for acquiring a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging area to the imaging device, and correcting the correction amount based on individual difference information for correcting individual differences in the luminance of the display panel. Effect of the Invention

[0008] According to the present invention, it is possible to obtain a correction amount that enables brightness control of an LED wall, which is an array of LED panels, so that display information can be captured with similar brightness regardless of the direction from which the LED wall is captured. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a system configuration. [Diagram 2]FIG. 2 is a block diagram showing an example of the hardware configuration of a computer device applicable to the image processing device 200. [Diagram 3] FIG. 2 is a block diagram showing an example of a more detailed functional configuration of the image processing device 200. [Figure 4] 6 is a flowchart of a process performed by the image processing device 200 to display display information on the LED wall. [Diagram 5] FIG. 11 is a diagram showing an example of a method for determining a correction region. [Figure 6] 10 is a flowchart showing details of the process in step S505. [Figure 7] FIG. 11 is a diagram for explaining the process in step S703. [Figure 8] FIG. 4 is a diagram showing an example of a table configuration. [Figure 9] FIG. 4 is a diagram showing an example of individual difference information. [Figure 10] FIG. 1 is a block diagram showing an example of a system configuration. [Figure 11] FIG. 2 is a block diagram showing an example of a more detailed functional configuration of the image processing device 200. [Figure 12] 6 is a flowchart of a process performed by the image processing device 200 to display display information on the LED wall. [Figure 13] 11 is a flowchart showing details of the process in step S1301. [Figure 14] FIG. 11 is a diagram for explaining the process in step S704. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] First, a configuration example of a system according to this embodiment will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a display device 300 that provides a display screen which is an array of display panels (array of display elements), an imaging device 100 that captures a part or all of the display screen, and an image processing device 200 that calculates a correction amount for the luminance of pixels on the display screen and outputs the correction amount together with display information to the display device 300. Note that the image processing device 200 and the display device 300, and the image processing device 200 and the imaging device 100 are configured to be able to communicate data with each other via a wired network, a wireless network, a network that is a combination of wired and wireless, or the like.

[0012] First, the display device 300 will be described. The display device 300 has a display board 302 and a display board 303 capable of displaying display information such as images and characters. In this embodiment, the display boards 302 and 303 are arranged side by side (adjacent to each other) to provide a display screen which is an arrangement of the display boards 302 and 303. In this embodiment, the display boards 302 and 303 are LED panels (arrangements of LED elements (pixels)), and in this case, the display device 300 (the display screen which is an arrangement of the display boards 302 and 303) is an LED wall. Note that, in FIG. 1, for the sake of simplicity, the number of display boards in the display device 300 (i.e., the number of display boards constituting the LED wall) is set to 2, but any number equal to or greater than 2 may be used.

[0013] Moreover, the display device 300 has a control unit 301. The control unit 301 controls various operations of the display device 300. For example, the control unit 301 divides the display information transmitted from the image processing device 200 into first display information to be displayed on the display board 302 and second display information to be displayed on the display board 303. In this embodiment, the display information transmitted from the image processing device 200 is divided into a left half and a right half, with the left half being the first display information and the right half being the second display information. Then, the control unit 301 causes the first display information to be displayed on the display board 302 and the second display information to be displayed on the display board 303.

[0014] For example, assume that the entire LED wall can display display information of A pixels vertically and B pixels horizontally. Here, the coordinates of the upper left corner of the LED wall are (0,0). In this case, display board 302 will be responsible for displaying the area in the display information whose upper left corner coordinates are (0,0) and whose lower right corner coordinates are (B / 2-1,A-1), and display board 303 will be responsible for displaying the area in the display information whose upper left corner coordinates are (B / 2,0) and whose lower right corner coordinates are (B-1,A-1). In this way, once the arrangement of the display boards is decided, it is determined which display board will be responsible for displaying which area of ​​the display information that can be displayed on the entire LED wall.

[0015] Based on the display information transmitted from the image processing device 200, the control unit 301 causes LED elements (pixels) at designated positions on the display boards 302 and 303 to emit light, thereby displaying the display information.

[0016] Next, the imaging device 100 will be described. The imaging device 100 is a camera that captures a part or the whole of the LED wall. The imaging device 100 may be a camera that captures moving images, or a camera that captures still images periodically or irregularly. The imaging device 100 may be a device equipped with such a camera (such as a tablet terminal device with a camera or a smartphone).

[0017] The imaging unit 101 includes a lens, a drive control unit that drives and controls the lens, an image sensor that photoelectrically converts light entering from the outside world through the lens into an image signal, and an image processing circuit that generates an image by performing various image processing based on the image signal.

[0018] The gyro sensor 102 measures its own position and orientation as the position and orientation of the imaging device 100 , respectively, and outputs position information indicating the position of the imaging device 100 and orientation information indicating the orientation of the imaging device 100 .

[0019] The transmitting / receiving unit 103 performs data communication with the image processing device 200. The transmitting / receiving unit 103 transmits, as shooting information, to the image processing device 200, the position information and attitude information output from the gyro sensor 102, the captured image generated by the imaging unit 101, and "focal length information indicating the focal length of the imaging device 100" and "sensor size information indicating the vertical and horizontal sizes of the image sensor" held by the imaging unit 101.

[0020] In addition, the imaging device 100 may further include a recording unit that records, for example, the captured image generated by the imaging unit 101 and information related to the captured image in its own memory or in a memory card that is removably attached to the imaging device 100.

[0021] Next, the image processing device 200 will be described. The image processing device 200 is a computer device such as a PC (personal computer), a tablet terminal device, or a smartphone, and executes various processes by communication with the imaging device 100 and the display device 300. The image processing device 200 specifies an imaging area in the LED wall that is imaged by the imaging device 100 based on parameters of the imaging device 100. The image processing device 200 acquires a correction amount of the luminance of the LED element (pixel) based on an angle from the LED element (pixel) in the imaging area to the imaging device 100, and corrects the correction amount based on information for correcting individual differences in the luminance of the display panel. The image processing device 200 outputs the corrected correction amount to the display device 300 together with display information. The display device 300 causes the LED element (pixel) to emit light at a luminance value corrected according to the display information output from the image processing device 200 according to the correction amount of the LED element (pixel). A more detailed functional configuration example of the image processing device 200 is shown in the block diagram of FIG. 3.

[0022] Next, the process performed by the image processing device 200 to display display information on the LED wall will be described with reference to the flowchart in Fig. 4. In step S501, the calculation unit 202 receives (acquires) shooting information transmitted from the imaging device 100. More specifically, the first input unit 220 included in the calculation unit 202 acquires the position information and orientation information transmitted from the imaging device 100, and the second input unit 221 included in the calculation unit 202 acquires the focal length information and sensor size information transmitted from the imaging device 100.

[0023] In step S503, the first calculation unit 222 of the calculation unit 202 acquires the arrangement information stored in the holding unit 201. The arrangement information is information indicating "the respective positions of the display boards 302 and 303" in the same coordinate system (hereinafter referred to as the shared coordinate system) as the position and attitude measured by the gyro sensor 102. The positions of the LED elements on the display boards 302 and 303 are known. Therefore, the positions of the "LED elements on the display boards 302 and 303" in the shared coordinate system can be obtained from the arrangement information and the positions of the LED elements on the display boards 302 and 303. The arrangement information may be information indicating the positions of the "LED elements on the display boards 302 and 303" in the shared coordinate system. Furthermore, the "normal vector of the LED wall" in the shared coordinate system can be obtained from the arrangement information and the positions of the LED elements on the display boards 302 and 303. The arrangement information may include a "normal vector of the LED wall" in the shared coordinate system.

[0024] In the following description, it is assumed that the positions of "the LED elements on each of display boards 302 and 303" in the shared coordinate system and "the normal vector of the LED wall" in the shared coordinate system are known based on the arrangement information.

[0025] In step S504, the first calculation unit 222 uses the shooting information acquired in step S501 and the arrangement information acquired in step S502 to obtain a correction area that is included in the angle of view range of the imaging device 100 on the LED wall. In other words, the correction area is an imaging area on the LED wall that is imaged by the imaging device 100.

[0026] Various methods can be applied to obtain the correction area, and any method may be used to obtain the correction area in this embodiment. An example of a method for obtaining the correction area will be described below with reference to FIG.

[0027] First, the first calculation unit 222 calculates the horizontal angle of view and the vertical angle of view of the image capture device 100 by a known calculation from the focal length information and the sensor size information. Next, the first calculation unit 222 calculates a vector 603 from the position 602 of the image capture device 100 indicated by the position information to the line of sight indicated by the attitude information. The vector 607 is a vector indicating the directly upward direction in a local coordinate system based on the attitude of the image capture device 100 indicated by the attitude information. Then, the first calculation unit 222 calculates a vector 604A from the position 602 toward the direction of "one end of the horizontal angle of view of the image capture device 100 in the local coordinate system with the vector 603 as the line of sight and the vector 607 as the directly upward direction". The first calculation unit 222 also calculates a vector 604B from the position 602 toward the direction of "the other end of the horizontal angle of view of the image capture device 100 in the local coordinate system with the vector 603 as the line of sight and the vector 607 as the directly upward direction". The first calculation unit 222 also calculates vector 605A pointing from position 602 in the direction of "one end of the vertical angle of view of image capture device 100 in a local coordinate system having vector 603 as the line of sight and vector 607 as the directly upward direction." The first calculation unit 222 also calculates vector 605B pointing from position 602 in the direction of "the other end of the vertical angle of view of image capture device 100 in a local coordinate system having vector 603 as the line of sight and vector 607 as the directly upward direction."

[0028] Furthermore, first calculation unit 222 obtains the vertical and horizontal directions of LED wall 500 in the shared coordinate system from the positions of "the LED elements on display boards 302 and 303" in the shared coordinate system, and calculates angle θA between the vertical direction and vector 607. This angle θA indicates the roll angle of imaging device 100 with respect to the line of sight. First calculation unit 222 calculates intersection position U with vector 605A on LED wall 500, intersection position D with vector 605B on LED wall 500, intersection position R with vector 604A on LED wall 500, and intersection position L with vector 604B on LED wall 500. Then, the first calculation unit 222 specifies, as the correction region, a rectangular region 606 (a rectangle shown by a dotted line in FIG. 5) surrounded by a line obtained by rotating a horizontal line passing through the intersection position U by θA, a line obtained by rotating a horizontal line passing through the intersection position D by θA, a line obtained by rotating a vertical line passing through the intersection position R by θA, and a line obtained by rotating a vertical line passing through the intersection position L by θA. Note that, in order to be able to handle a sudden movement of the imaging device 100, a region including the region 606 specified by the above method may be set as the correction region.

[0029] In step S505, the second calculation unit 223 of the calculation unit 202 calculates the correction amount for correcting the luminance value of each LED element (pixel) in the correction area calculated in step S504. The LED elements have an R (red) element, a G (green) element, and a B (blue) element, and in step S505, the second calculation unit 223 calculates the correction amount for correcting the luminance value of the R element, the correction amount for correcting the luminance value of the G element, and the correction amount for correcting the luminance value of the B element. Details of the process in step S505 will be described later.

[0030] In step S506, the generation unit 203 in the calculation unit 202 generates display information to be displayed on the LED wall based on the image and character data stored in the storage unit 201. The display information includes information to be displayed in the correction area and information to be displayed outside the correction area. In step S507, the generation unit 203 outputs the display information generated in step S506 and the correction amount calculated in step S505 to the display device 300.

[0031] Next, the details of the process in step S505 above will be described with reference to the flowchart in FIG. 6. In step S703, the second calculation unit 223 calculates an angle from the position of each LED element (pixel) in the correction area toward the position of the imaging device 100 indicated by the position information. The process in step S703 will be described with reference to FIG. 7. In FIG. 7, an area 800 is a part of the LED wall, and includes the area 606, which is the correction area. Each rectangle in the area 800 represents an LED element (pixel), and the coordinates of the rectangle 801 in the upper left corner of the area 800 are (x, y). In this case, the coordinates of the rectangle in the upper left corner of the area 606 are (x+3, y+1), and the coordinates of the rectangle in the lower right corner are (x+5, y+3).

[0032] Here, when determining the angle corresponding to the LED element (pixel) at the coordinate (x+3, y+1), the second calculation unit 223 determines the angle between the normal vector 803 of the LED wall and the vector 802 pointing from the coordinate (x+3, y+1) toward the position 602 as the angle corresponding to the LED element (pixel) at the coordinate (x+3, y+1). The second calculation unit 223 performs such processing for each LED element (pixel) included in the area 606, thereby being able to determine the angle corresponding to each LED element (pixel).

[0033] The angle corresponding to the LED element (pixel) at the coordinates (x+3, y+1) is expressed by two components. In the following, as shown in FIG. 7, the horizontal direction of the LED wall is the x-axis direction, the vertical direction of the LED wall is the y-axis direction, and the direction of the normal vector 803 of the LED wall (each LED element) is the z-axis direction. Here, the normal vector 803 (unit vector) = (a1, a2, a3), and the vector 802 (unit vector) = (b1, b2, b3). At this time, the second calculation unit 223 can obtain the angle (θXZ, θXY) corresponding to the LED element (pixel) at the coordinates (x+3, y+1) by calculating the following formulas (1) and (2).

[0034]

number

[0035] In step S704, the second calculation unit 223 obtains the "luminance correction amount" corresponding to the angle calculated in step S703 for each pixel in the correction area. Here, the storage unit 201 has a table shown in FIG.

[0036] In the table of Fig. 8, the correction amount of the luminance value of the LED element (pixel) corresponding to each angle of -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees is registered, which includes the correction amount of the luminance value of R (red) (R element correction amount), the correction amount of the luminance value of G (green) (G element correction amount), and the correction amount of the luminance value of B (blue) (B element correction amount). This table is a table showing the correction amount according to the angle, which is expressed as a signed angle from the left or right of the LED element and how obliquely the LED element is imaged by the imaging device 100, with the angle to the LED element facing the imaging device 100 being 0 degrees. In Fig. 8, the angle from the LED element to the imaging device 100 on the left side is expressed as a negative angle, and the angle from the LED element to the imaging device 100 on the right side is expressed as a positive angle.

[0037] Here, Fig. 8 shows the amount of correction according to the angle with the Z axis (normal vector 803) on the XZ plane. However, in reality, as shown in Fig. 14, a correction amount according to the angle between the normal vector 803 and vectors from the position 1599 of the LED element to various positions on a hemisphere centered on the center 1599 is required.

[0038] In this hemisphere, a point on the imaging device 100 directly facing the LED element at position 1599, that is, a point where the angle is 0 degrees, is point 1503. In this hemisphere, the left half is expressed as a negative angle and the right half is expressed as a positive angle with respect to normal vector 803 as the center. For example, among the points on an arc (called a basic arc) where Y=0 in the hemisphere, a point where the angle is "-30 degrees" is a point 30 degrees to the left of normal vector 803, and is therefore point 1504. The table in FIG. 8 is a table in which correction amounts corresponding to the respective points on the basic arc where the angles formed with normal vector 803 are -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees are registered.

[0039] In order to prepare correction amounts according to the angles between the normal vector 803 and vectors from the position 1599 to various positions on the hemisphere, the correction amounts corresponding to "points on the rotated basic arc where the angle with the normal vector 803 is -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees" when the basic arc is rotated around the Z axis by Δ degrees, 2Δ degrees, ..., 180 degrees are obtained by interpolation processing. For example, the correction amount corresponding to the point 1506 (the point on the arc where X=0 on the hemisphere where the angle with the normal vector 803 is "-80 degrees") is the average value of the correction amount CA1 corresponding to the point on the basic arc where the angle with the normal vector 803 is "-80 degrees" and the correction amount CA2 corresponding to the point on the basic arc where the angle with the normal vector 803 is "80 degrees".

[0040] The correction amount CD corresponding to the "point at which the angle formed with the normal vector 803 is -80 degrees" on the rotated basic arc obtained by rotating the basic arc by S degrees around the Z axis can be calculated using, for example, the following formula.

[0041] CD = (180-S)xCA / 180+SxCB / 180 By performing such a calculation for S=Δ degrees, 2Δ degrees, ..., 180 degrees, it is possible to obtain the correction amount corresponding to "the point where the angle formed with the normal vector 803 is -80 degrees" for each of Δ degrees, 2Δ degrees, ..., 180 degrees. Then, such a calculation is performed for "the point where the angle formed with the normal vector 803 is θ degrees" (θ=-80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, 80 degrees).

[0042] Through such processing, the second calculation unit 223 obtains, as reference information, the amount of correction according to the angle (θXZ, θXY) between the vector from the LED element position 1599 to various positions on the hemisphere and the normal vector 803.

[0043] In this embodiment, a table is used in which correction amounts corresponding to angles of -80 degrees, -60 degrees, -45 degrees, -30 degrees, 0 degrees, 30 degrees, 45 degrees, 60 degrees, and 80 degrees are registered, but the number of registered angles is not limited to a specific number. For example, a table in which correction amounts corresponding to a larger number of angles are registered may be used.

[0044] In addition, in this embodiment, a process is performed to obtain the correction amount corresponding to various angles from the table in FIG. 8, but such a process may be performed in advance to obtain the correction amount corresponding to various angles and register it as data in the storage unit 201.

[0045] The second calculation unit 223 then refers to the reference information and acquires the amount of correction of the luminance value corresponding to the angle obtained in step S703 for each LED element (pixel) in the correction area. When the reference information is generated using the table in FIG. 8, the amount of correction is not obtained for angles of 80 degrees or more. Therefore, for LED elements (pixels) whose angles obtained in step S703 are 80 degrees or more, the amount of correction corresponding to 80 degrees is acquired. Also, if the angle obtained in step S703 is not registered in the reference information, an interpolated correction amount obtained from the correction amounts corresponding to angles close to the angle obtained in step S703 among the angles registered in the reference information is acquired as the amount of correction corresponding to the angle obtained in step S703.

[0046] In addition, in this embodiment, the LED wall has been described as having LED elements having R elements, G elements, and B elements, but in addition to these, it may have LED elements of other colors, such as white LEDs (W). In this case, in addition to the correction amounts for R, G, and B, it is necessary to obtain correction amounts corresponding to LED elements of other colors.

[0047] Next, in step S705, second calculation unit 223 performs a correction process on the correction amount acquired in step S704 to reduce the influence caused by individual differences in the luminance of display board 302 and display board 303. Due to factors such as variations in precision of parts during manufacturing, LED panels may output images with different brightnesses even when the same video signal is applied. For this reason, in this embodiment, correction coefficients for R, G, and B are registered in storage unit 201 as individual difference information for correcting individual differences in the luminance of LED elements in each of display boards 302 and 303. An example of individual difference information is shown in FIG. 9.

[0048] As shown in Figure 9, the individual difference information includes, for each of display board 302 and display board 303, the position of the display board in the shared coordinate system (panel position), the correction amount of R (correction amount R), the correction amount of G (correction amount G), and the correction amount of B (correction amount B).

[0049] Second calculation unit 223 performs correction by multiplying the correction amounts of R, G, and B obtained in step S704 for LED elements (pixels) in the correction area that belong to display board 302 by the correction amounts R, G, and B corresponding to the positions of display board 302 indicated by the arrangement information. Similarly, second calculation unit 223 performs correction by multiplying the correction amounts of R, G, and B obtained in step S704 for LED elements (pixels) in the correction area that belong to display board 303 by the correction amounts R, G, and B corresponding to the positions of display board 303 indicated by the arrangement information.

[0050] Then, in step S706, the second calculation unit 223 outputs to the generation unit 203 the “correction amount of each LED element (pixel) in the correction area” corrected in step S705.

[0051] Next, a description will be given of the operation of the display device 300 that has acquired the display information and the correction amount from the image processing device 200. The control unit 301 obtains the luminance value of each pixel in the correction area in the display information acquired from the image processing device 200 by multiplying the luminance value of the pixel by the correction amount acquired from the image processing device 200 for the pixel.

[0052] For example, assume that pixel P at coordinates (a, b) in the correction area is a pixel on display panel 302, that the luminance value of pixel P is (R, G, B) = (100, 100, 150), and that the correction amount acquired in step S704 for pixel P is (1.12, 1.11, 1.19). At this time, according to the individual difference information in Fig. 9, since the panel position of display panel 302 is (1, 1), the correction amount R, correction amount G, and correction amount B corresponding to panel position (1, 1) are 1.01, 1.02, and 1.01, respectively.

[0053] In this case, the control unit 301 multiplies the correction amount of R "1.12" obtained for pixel P in step S704 by the correction amount R "1.01" to obtain a correction amount of "1.1312" for pixel P from the image processing device 200.

[0054] Similarly, the control unit 301 multiplies the G correction amount "1.11" obtained for pixel P in step S704 by the correction amount R "1.02" to obtain a correction amount of "1.1322" from the image processing device 200 as the correction amount of G for pixel P.

[0055] Similarly, the control unit 301 multiplies the correction amount of B "1.19" obtained for pixel P in step S704 by the correction amount R "1.01" to obtain a correction amount of "1.2019" as the correction amount of B for pixel P from the image processing device 200.

[0056] Then, the control unit 301 multiplies the R luminance value of pixel P, "100", by the R correction amount of pixel P, "1.1312", to obtain "113" (rounded off to the first decimal place) as the corrected R luminance value of pixel P.

[0057] Similarly, the control unit 301 multiplies the G luminance value of pixel P, "100", by the G correction amount of pixel P, "1.1322", to obtain "113" (rounded to the first decimal place) as the corrected G luminance value of pixel P.

[0058] Similarly, the control unit 301 multiplies the B luminance value of pixel P, "150", by the correction amount of B of pixel P, "1.2019", to obtain "180" (rounded to the first decimal place) as the corrected B luminance value of pixel P.

[0059] The control unit 301 then controls the light emission of the LED elements so that the R element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "113", the G element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "113", and the B element of the LED element corresponding to pixel P emits light at a brightness corresponding to the brightness value "180".

[0060] In this way, in this embodiment, regardless of the relative positional relationship between the imaging device and the LED wall or the variation in brightness between the LED panels on the LED wall, it is possible to obtain an image of the LED wall with a constant brightness within the field of view of the imaging device.

[0061] In this embodiment, the image processing device 200 outputs the display information and the correction amount to the display device 300, and the display device 300 causes the LED elements to emit light based on the luminance value obtained by correcting the luminance value of the display information in accordance with the correction amount. However, the image processing device 200 may perform the above-mentioned luminance value correction using the display information and the correction amount, and output the corrected luminance value to the display device 300. In this case, the display device 300 controls the emission of the LED elements in accordance with the luminance value acquired from the image processing device 200.

[0062] In the present embodiment, the case has been described in which the amount of correction corresponding to the angle of each LED element (pixel) is obtained using the reference information, and the amount of correction is output to the display device 300 as display information.

[0063] However, such reference information may be stored in a memory device such as the HDD 405 or the external storage device 409, or may be transmitted to an external device via the network I / F 412. With this configuration, when obtaining the amount of correction corresponding to the angle of each LED element (pixel) from the next time onwards, the reference information stored in the memory device or the external device may be obtained and used. Also, reference information generated by an external device may be obtained and stored in the HDD 405, and read out and used as necessary.

[0064] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. A configuration example of a system according to this embodiment is shown in the block diagram of Fig. 10. In the configuration shown in Fig. 10, the image processing device 200 in the system shown in Fig. 1 has a storage unit 1101.

[0065] A more detailed example of the functional configuration of the image processing device 200 is shown in the block diagram of Fig. 11. The configuration shown in Fig. 11 is the same as the configuration shown in Fig. 3 except that a holding unit 1101 is added and a third calculation unit 1222 is provided instead of the first calculation unit 222. The third calculation unit 1222 performs the above-mentioned processing as the processing performed by the first calculation unit 222, and also obtains information on peripheral light loss in the correction area.

[0066] Next, the process performed by the image processing device 200 to display the display information on the LED wall will be described with reference to the flowchart in Fig. 12. In Fig. 12, the same step numbers are used for the same process steps as those shown in Fig. 4, and the description of these process steps will be omitted. In this embodiment, the processes of steps S503 and S504 are performed by the third calculation unit 1222.

[0067] In step S1301, the third calculation unit 1222 acquires "peripheral light falloff information indicating peripheral light falloff for a lens" stored in the storage unit 1211. The peripheral light falloff information is information for correcting peripheral light falloff corresponding to a lens attached to the image capture device 100, and is, for example, a map that stores "correction coefficients for correcting peripheral light falloff" for each pixel of an image captured by the image capture device 100.

[0068] Here, the peripheral light falloff information will be described. The peripheral light falloff correction information is a map of correction coefficients for correcting peripheral light falloff, which is a decrease in brightness as the distance from the center of the angle of view increases, for an image captured by the imaging device 100.

[0069] The third calculation unit 1222 then maps the peripheral light falloff information to the correction area to obtain a correction coefficient corresponding to each pixel in the correction area. The "correction coefficient corresponding to each pixel in the correction area" becomes the "peripheral light falloff information of the correction area."

[0070] When mapping the peripheral light falloff information to the correction area, there is a possibility that there will be some parts in the peripheral light falloff information after mapping where no correction coefficient exists, so in such parts, a correction coefficient that is interpolated by a well-known interpolation technique using the peripheral correction coefficients may be set. Note that the method for acquiring the correction coefficients corresponding to each pixel in the correction area is not limited to a specific method.

[0071] Furthermore, the peripheral light falloff information is not limited to a map that holds correction coefficients for each pixel of an image captured by the imaging device 100, but may be, for example, a calculation formula that can be applied to correct peripheral light falloff of pixels in a captured image.

[0072] The storage unit 1101 may store peripheral light falloff information for each type of lens. In this case, the third calculation unit 1222 acquires "identification information of the lens attached to the image capture device 100" managed by the image capture unit 101, and acquires peripheral light falloff information corresponding to the identification information.

[0073] Then, the second calculation unit 223 obtains a correction amount by correcting the correction amount for each pixel in the correction area based on the individual difference information, in the same manner as in the first embodiment. Then, the second calculation unit 223 obtains a correction amount by correcting the corrected correction amount based on the correction coefficient acquired by the third calculation unit 1222.

[0074] Details of the process in step S1301 will be described with reference to the flowchart in Fig. 13. In Fig. 13, the same process steps as those shown in Fig. 6 are given the same step numbers, and descriptions of those process steps will be omitted.

[0075] In step S1401, the third calculation unit 1222 obtains a correction coefficient corresponding to each pixel in the correction area. The second calculation unit 223 multiplies the correction amount for each pixel corrected in step S705 by the correction coefficient in the correction area that corresponds to the pixel to obtain a correction amount after correction. In step S706, the second calculation unit 223 outputs the "correction amount for each pixel in the correction area" corrected in step S1401 to the generation unit 203.

[0076] In this way, in this embodiment, regardless of the relative positional relationship between the imaging device and the LED wall or the variation in brightness between the LED panels on the LED wall, it is possible to obtain an image of the LED wall with a constant brightness within the field of view of the imaging device and with peripheral light falloff of the lens corrected.

[0077] [Third embodiment] In the first and second embodiments, the individual difference information is created in advance and registered in storage unit 201. However, before using this system, individual differences in the luminance of the LED elements in each of display boards 302 and 303 may be measured, and individual difference information may be generated based on the measurement results and registered in storage unit 201.

[0078] For example, a second generating unit (not shown) may be provided in the image processing device 200, and the second generating unit may generate individual difference information using a captured image of "the image displayed on the display board 302" captured by the imaging device 100 and a captured image of "the image displayed on the display board 303" captured by the imaging device 100. More specifically, the same test chart is displayed on each of the display boards 302 and 303. The test chart may be, for example, a monochromatic chart using signal values ​​such as gray, red, green, and blue that make the LED panel uniform in color. The imaging device 100 then captures the display board 302 in a position and orientation that faces the center of the screen of the display board 302 on which such a test chart is displayed, and obtains a captured image of the display board 302. Similarly, the imaging device 100 captures the display board 303 in a position and orientation that faces the center of the screen of the display board 303 on which such a test chart is displayed, and obtains a captured image of the display board 303.

[0079] The second generation unit obtains an average value R1 of pixel values ​​of R in an image region of "the test chart displayed on display board 302" from the captured image of display board 302. The second generation unit also obtains an average value R2 of pixel values ​​of R in an image region of "the test chart displayed on display board 303" from the captured image of display board 303. Then, the second generation unit uses R1 and R2 to determine a correction amount R of display board 302 and a correction amount R of display board 303, and registers them in the individual difference information. For example, the second generation unit registers R2 / R1 as the correction amount R of display board 302 and R1 / R2 as the correction amount R of display board 303 in the individual difference information.

[0080] Furthermore, the second generation unit obtains an average value G1 of pixel values ​​of G in an image region of "the test chart displayed on display board 302" from the captured image of display board 302. Furthermore, the second generation unit obtains an average value G2 of pixel values ​​of G in an image region of "the test chart displayed on display board 303" from the captured image of display board 303. Then, the second generation unit uses G1 and G2 to determine the correction amount G of display board 302 and the correction amount G of display board 303, and registers them in the individual difference information. For example, the second generation unit registers G2 / G1 as the correction amount G of display board 302 and G1 / G2 as the correction amount G of display board 303 in the individual difference information.

[0081] Furthermore, the second generation unit obtains an average value B1 of pixel values ​​of B in an image region of "the test chart displayed on display board 302" from the captured image of display board 302. Furthermore, the second generation unit obtains an average value B2 of pixel values ​​of B in an image region of "the test chart displayed on display board 303" from the captured image of display board 303. Then, the second generation unit uses B1 and B2 to determine the correction amount B of display board 302 and the correction amount B of display board 303, and registers them in the individual difference information. For example, the second generation unit registers B2 / B1 as the correction amount B of display board 302 and B1 / B2 as the correction amount B of display board 303 in the individual difference information.

[0082] [Fourth embodiment] Each functional unit of the image processing device 200 shown in Figs. 1, 3, 10, and 11 may be implemented by hardware, or each functional unit except for the storage units 201 and 1101 may be implemented by software (computer program). In the latter case, a computer device capable of executing such a computer program is applicable to the image processing device 200. An example of the hardware configuration of a computer device applicable to the image processing device 200 will be described with reference to the block diagram of Fig. 2. Applicable examples of such computer devices include PCs (personal computers), WSs (workstations), smartphones, and tablet terminal devices.

[0083] The CPU 401 executes various processes using computer programs and data stored in the RAM 402 and the ROM 403. As a result, the CPU 401 controls the operation of the entire computer device, and executes or controls the various processes described as the processes performed by the image processing device 200.

[0084] The RAM 402 has areas for storing computer programs and data loaded from the ROM 403, the HDD (hard disk drive) 405, and the external storage device 409. The RAM 402 also has areas for storing various information acquired from the imaging device 100 via the network I / F 412. The RAM 402 also has a work area used when the CPU 401 executes various processes. In this way, the RAM 402 provides various areas as appropriate.

[0085] In addition to the CPU 401 and RAM 402, a GPU for rendering display information to be displayed on the LED wall (e.g., CG rendering) and a VRAM for storing image and text data (CG data, scene data, etc.) may also be provided.

[0086] The ROM 403 stores setting data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and the like.

[0087] The auxiliary storage I / F 404 is an interface for connecting the HDD 405 to the system bus 408. The HDD 405 stores an OS (operating system), computer programs and data for causing the CPU 401 to execute or control the various processes described above as processes performed by the image processing device 200, and the like. The computer programs and data stored in the HDD 405 are loaded into the RAM 402 as appropriate under the control of the CPU 401, and become targets for processing by the CPU 401. Note that in addition to or instead of the HDD 405, various storage devices such as an optical disk drive and a flash memory can also be used.

[0088] The input I / F 406 is, for example, a serial bus interface such as USB or IEEE1394, and is an interface for connecting an input device 411 and an external storage device 409 to the image processing device 200 .

[0089] The input device 411 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can input various instructions to the CPU 401 by operating it.

[0090] The external storage device 409 is a storage device that is detachable from the image processing device 200, and is a storage device such as a hard disk drive, a memory card, a CF card, an SD card, a USB memory, etc. Some or all of the computer programs and data described as being stored in the HDD 405 or the like may be stored in the external storage device 409.

[0091] The above-mentioned storage unit 201 and storage unit 1101 can be implemented using one or more memory devices selected from the RAM 402, the ROM 403, the HDD 405, and the external storage device 409 in the case of FIG.

[0092] The output I / F 407, like the input I / F 406, is a serial bus interface such as USB or IEEE1394, and is an interface for connecting the monitor 410 and the display device 300 to the image processing device 200. The output I / F 407 may be a video output terminal such as DVI or HDMI (registered trademark).

[0093] The monitor 410 is a display device having a liquid crystal screen or a touch panel screen, and displays the results of processing by the CPU 401 as images, characters, etc. For example, the CPU 401 causes the monitor 410 to display data processed by the CPU 401 (for example, real-time arrangement status of the imaging device 100 and the display device 300, etc.).

[0094] The network I / F 412 is a connector or the like for connecting to a network such as Ethernet, and is an interface for connecting the imaging device 100 to the image processing device 200.

[0095] The CPU 401, RAM 402, ROM 403, auxiliary storage I / F 404, input I / F 406, output I / F 407, and network I / F 412 are all connected to a system bus 408. Note that the configuration shown in Fig. 2 is an example of a hardware configuration of a computer device applicable to the image processing device 200, and can be changed / modified as appropriate.

[0096] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0097] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0098] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0099] The invention of this specification includes the following image processing device, image processing method, and computer program.

[0100] (Item 1) A specifying means for specifying an imaging area to be imaged by an imaging device on a display screen, which is an array of display panels, based on parameters of the imaging device; a correction means for acquiring a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging region to the imaging device, and correcting the correction amount based on individual difference information for correcting individual differences regarding luminance of a display panel; An image processing device comprising:

[0101] (Item 2) 2. The image processing device according to item 1, wherein the specifying means specifies, on the display screen, an area included in a range of an angle of view of the imaging device as the imaging area, based on parameters of the imaging device.

[0102] (Item 3) The correction means is an angle between a vector from the pixel to the imaging device and a normal vector of the display screen is calculated, and a correction amount corresponding to the angle is obtained as a correction amount for the luminance of the pixel; The amount of correction of the luminance of the pixel is multiplied by a correction coefficient for correcting individual differences in the luminance of the display panel to correct the amount of correction. 3. The image processing device according to item 1 or 2.

[0103] (Item 4) The correction means corrects the correction amount by multiplying the correction amount by a correction coefficient for correcting individual differences in luminance of the display panel and a correction coefficient for correcting peripheral light loss of the lens. 4. The image processing device according to any one of items 1 to 3,

[0104] (Item 5) moreover, The image processing device according to any one of items 1 to 4, further comprising an output unit that outputs display information to be displayed on the display screen and the amount of correction performed by the correction unit to a display device that provides the display screen.

[0105] (Item 6) moreover, The image processing device according to any one of items 1 to 5, further comprising a generating unit that generates the individual difference information based on pixel values ​​of an image area of ​​a test chart in a captured image of the test chart displayed on each display panel.

[0106] (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the display panel is an LED panel, and the pixels are LED elements in the LED panel.

[0107] (Item 8) An imaging device; An image processing device; A display device that provides a display screen that is an array of display panels; A system having The image processing device includes: a specifying means for specifying an imaging area on the display screen that is imaged by the imaging device based on parameters of the imaging device; a correction means for acquiring a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging region to the imaging device, and correcting the correction amount based on information for correcting individual differences in luminance of a display panel; an output means for outputting display information and the amount of correction performed by the correction means to the display device; Equipped with The display device includes: a control unit for correcting the luminance value of each pixel in the imaging area of ​​the display information according to the correction amount of the pixel and displaying the corrected luminance value on the display screen; A system characterized by:

[0108] (Item 9) An image processing method performed by an image processing device, comprising: a specifying step in which a specifying means of the image processing device specifies an imaging area to be imaged by an imaging device on a display screen which is an array of display panels, based on parameters of the imaging device; a correction step in which a correction means of the image processing device acquires a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging area to the imaging device, and corrects the correction amount based on individual difference information for correcting individual differences related to luminance of a display panel; An image processing method comprising:

[0109] (Item 10) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 7.

[0110] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0111] 100: Imaging device 101: Imaging unit 102: Gyro sensor 103: Transmitting / receiving device 200: Image processing device 201: Storage unit 202: Calculation unit 203: Generation unit 300: Display device 301: Control unit 302: Display panel 303: Display panel

Claims

1. A specifying means for specifying an imaging area to be imaged by an imaging device on a display screen, which is an array of display panels, based on parameters of the imaging device; a correction means for acquiring a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging region to the imaging device, and correcting the correction amount based on individual difference information for correcting individual differences regarding luminance of a display panel; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the specifying means specifies, on the basis of parameters of the imaging device, an area on the display screen that is included in a range of an angle of view of the imaging device as the imaging area.

3. The correction means is an angle between a vector from the pixel to the imaging device and a normal vector of the display screen is calculated, and a correction amount corresponding to the angle is obtained as a correction amount for the luminance of the pixel; The amount of correction of the luminance of the pixel is multiplied by a correction coefficient for correcting individual differences in the luminance of the display panel to correct the amount of correction.

2. The image processing device according to claim 1,

4. The correction means corrects the correction amount by multiplying the correction amount by a correction coefficient for correcting individual differences in luminance of the display panel and a correction coefficient for correcting peripheral light loss of the lens.

2. The image processing device according to claim 1,

5. moreover, 2. The image processing apparatus according to claim 1, further comprising an output unit that outputs display information to be displayed on the display screen and the amount of correction performed by the correction unit to a display device that provides the display screen.

6. moreover, 2. The image processing apparatus according to claim 1, further comprising a generating unit that generates the individual difference information based on pixel values ​​of an image area of ​​a test chart in a captured image of the test chart displayed on each display panel.

7. 2. The image processing apparatus according to claim 1, wherein the display panel is an LED panel, and the pixels are LED elements in the LED panel.

8. An imaging device; An image processing device; A display device that provides a display screen that is an array of display panels; A system having The image processing device includes: a specifying means for specifying an imaging area on the display screen that is imaged by the imaging device based on parameters of the imaging device; a correction means for acquiring a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging region to the imaging device, and correcting the correction amount based on information for correcting individual differences in luminance of a display panel; an output means for outputting display information and the amount of correction performed by the correction means to the display device; Equipped with The display device includes: a control unit for correcting the luminance value of each pixel in the imaging area of ​​the display information according to the correction amount of the pixel and displaying the corrected luminance value on the display screen; A system characterized by:

9. An image processing method performed by an image processing device, comprising: a specifying step in which a specifying means of the image processing device specifies an imaging area to be imaged by an imaging device on a display screen which is an array of display panels, based on parameters of the imaging device; a correction step in which a correction means of the image processing device acquires a correction amount for a luminance value of a pixel based on an angle from the pixel in the imaging area to the imaging device, and corrects the correction amount based on individual difference information for correcting individual differences related to luminance of a display panel; An image processing method comprising:

10. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 7.

Images