Image processing apparatus and control method thereof, and program

Abstract

To display a natural background image on a display device.SOLUTION: According to the present invention, an image processing apparatus that generates an image projected on a display device which has a display screen having one or more display panels constituted of display elements each having a brightness property depending on an angle is provided. The image processing apparatus comprises: an acquisition unit that acquires information associated with a position and an attitude of an imaging device for imaging a subject having a video displayed on a display screen of a display device as a background; a calculation unit that calculates a correction coefficient corresponding to each display element on the basis of a display position of the display device and information acquired by the acquisition unit; and a correction unit that corrects, using the correction coefficient calculated by the calculation unit, each pixel in image data to be displayed on the display device. The image processing apparatus is characterized in that of outputting, to the display device, the image data after being corrected by the correction unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to image processing technology in virtual production that combines display devices such as LED walls with shooting devices such as cameras. [Background technology]

[0002] Conventionally, there are known techniques for correcting an image on a display monitor according to the position of an observer (Patent Documents 1 and 2). The techniques described in Patent Documents 1 and 2 acquire the position of the observer by an imaging device such as a camera, and correct the image on the display monitor based on the acquired position information.

[0003] Recently, in the field of video production, a technique called virtual production has become popular in which images are displayed on a display device made up of multiple LED panels, such as LED walls, and the images are then captured by a camera. In virtual production, the camera's movement and line of sight are measured in real time, and the images displayed on the LED wall within the camera's field of view are changed in real time. By capturing this with a camera, the technique allows for footage that makes it seem as if the real thing were actually there. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-42804 A [Patent Document 2] JP 2009-128381 A Summary of the Invention [Problem to be solved by the invention]

[0005] The display elements (LED elements) that make up the pixels of LED walls, which are often used in virtual production, have directional characteristics. Therefore, the brightness is high when the LED wall is observed from the front, but the brightness is low when observed from an oblique direction. Also, as mentioned above, LED walls in virtual production are often composed of multiple LED panels. However, the display monitor that is the subject of correction in conventional technology is a display device composed of a single panel, so there is a problem that an unnatural background image is displayed on an LED wall composed of multiple LED panels.

[0006] The present invention provides a technique that allows a subject to be imaged while a natural background image is displayed on a display device. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image processing device according to the present invention has the following arrangement. An image processing device that generates an image to be displayed on a display device having one or more display panels arranged as a display screen, the display panels being made up of display elements having brightness characteristics that depend on an angle, comprising: an acquisition means for acquiring information regarding a position and an attitude of an imaging device for capturing an image of a subject against a background of an image displayed on a display screen of the display device; a calculation means for calculating a correction coefficient for each display element based on the display position of the display device and the information acquired by the acquisition means; a correction means for correcting each pixel in the image data to be displayed on the display device by the correction coefficient calculated by the calculation means, The image data corrected by the correction means is output to the display device. Effect of the Invention

[0008] According to the present invention, it is possible to capture an image of a subject while displaying a natural background image on a display device. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a specific hardware configuration of the image processing device. [Diagram 3] FIG. 2 is a detailed functional configuration diagram of an image processing apparatus according to an embodiment. [Figure 4] 4 is a flowchart showing basic processing of the image processing apparatus according to the embodiment. [Diagram 5] 5A to 5C are diagrams for explaining how to obtain a correction area in the embodiment. [Figure 6] 6 is a flowchart showing a process of calculating a pixel correction amount within a correction area on a display screen of a display device in an embodiment. [Figure 7] 11 is a conceptual diagram illustrating how a specific pixel position is calculated from a correction area. [Figure 8] 11 is a flowchart showing the flow of a correction amount allocation process. [Figure 9] 1 is a table showing information on the angular variation characteristics of the output brightness of an LED element. [Figure 10] A table showing the variability between LED panels. [Figure 11] FIG. 13 is a conceptual diagram showing a method for complementing deformation characteristic information. [Figure 12] FIG. 13 is a conceptual diagram showing a method for calculating the angle between the imaging device and an LED element. [Figure 13] 6 is a diagram showing the correspondence between lighting devices and exposure amount correction information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] First, the image processing system of this embodiment has an imaging device, an image processing device, and a display device (LED wall). The imaging device has a transmission / reception device. The transmission / reception device acquires focal length information of the lens of the camera, position information of the camera obtained from a gyro sensor, and information indicating the line of sight, and transmits the acquired information to the image processing device as shooting information. The image processing device stores and holds display device information such as information on the arrangement of the display device and information on individual variations of each display LED panel (display panel) arranged on the display device. The image processing device generates correction information for the image to be displayed based on the shooting information and display device information received from the imaging device. The image processing device then judges whether the generated correction information is within the brightness control range of the LED panel, and if it is outside the brightness control range of the LED panel, corrects the image signal and transmits the corrected production to the LED panel. A more detailed explanation will be given below.

[0012] 1 is a configuration diagram of a video system according to the first embodiment. As shown in the figure, the system includes an imaging device 100, an image processing device 200, a display device (LED wall) 300, and a lighting device 400.

[0013] The imaging device 100 has a function of transmitting the above-mentioned shooting information to the image processing device 200. The imaging device 100 also has a function of shooting a subject including the display device 300 and recording the shot video. For this reason, the imaging device 100 is equipped with a sensor, a focus mechanism, a recording medium for recording video, and the like, which are not shown.

[0014] The image processing device 200 generates and transmits an image to be displayed on the display device (LED wall) 300 based on the shooting information (lens focal length information, position information, and information indicating the line of sight direction) received from the imaging device 100. The details of the process and configuration of the image processing device 200 will be described later.

[0015] The display device 300 is a large display device made up of multiple display panels (LED panels). In this embodiment, for the sake of simplicity, it is assumed that the display device 300 is made up of two LED panels 301 and 302. Of course, there is no limit to the number of LED panels that make up the display device 300, and it may be three or more. The arrangement of the LED panels is not limited to the left-right direction, and they may be arranged two-dimensionally, up-down, left-right, or both.

[0016] The lighting device 400 is mainly used to illuminate a main subject (performer) standing in front of a background displayed on the LED wall 300 during shooting. In this embodiment, the lighting device 400 controls the lighting with output values ​​ranging from 0 to 100. The image processing device 209 sets the lighting device 400 to an output of "50", which is the central value, as the initial state.

[0017] The imaging device 100 includes a lens 101, a gyro sensor 102, a transmitting / receiving device 103, a camera engine 104, and an imaging unit 105. The lens 101 includes a plurality of lenses such as a zoom lens and a focus lens. The lens 101 forms an image of a subject (performer) against the background of an image displayed by the display device 300 on an imaging surface of an image sensor of the imaging unit 105. Under the control of the camera engine 104, the imaging unit 105 converts the formed image into an electrical signal and records it on a recording medium (not shown). It is assumed that the imaging unit 105 captures images at a frame rate of, for example, 30 frames per second. The camera engine 104 changes the shutter speed and ISO sensitivity per frame of the imaging device. The camera engine 104 also performs development processing of the image captured by the imaging unit 105. The gyro sensor 103 detects the position and orientation of the imaging device 100 (and therefore the optical axis direction of the lens = line of sight direction) and supplies the detected information to the transmitting / receiving device 103 .

[0018] The transmitting / receiving device 103 compiles the current focal length of the lens 101, sensor size information of the imaging unit 105, and position and orientation information obtained by the gyro sensor (information including the position and line of sight direction of the imaging device 100) as shooting information. Then, the transmitting / receiving device 103 transmits the shooting information to the image processing device 200. Furthermore, the transmitting / receiving device 103 changes the aperture value for the imaging unit 105 based on the information from the image processing device 200.

[0019] The image processing device 200 includes a display device information storage unit 201 , a correction information calculation unit 202 , and a video generation unit 203 .

[0020] The display device information storage unit 201 has a role of storing display position information of the display device 300 and characteristic information of the display device. The display position information refers to arrangement information indicating the position of the display board (LED panel) constituting the display device 300. The arrangement information is expressed in the same coordinate system as the camera position information and line of sight information, and the angle between the line of sight of the imaging device 100 and the LED wall 300 can be uniquely determined and calculated from the arrangement information and the shooting information. In addition to the above-mentioned display position information, the display device information storage unit 201 in this embodiment also has a role of storing individual difference information indicating individual differences of the LED panels and CG data to be displayed on the display device 300. The CG data refers to CG scene data for rendering CG. The CG scene data is video data rendered by the video generation unit 203.

[0021] The correction information calculation unit 202 generates correction information for the display device 300 based on the shooting information transmitted from the imaging device 100 and the arrangement information stored in the display device information storage unit 201. Details will be described later.

[0022] The image generating unit 203 renders the CG scene data stored in the display device information storage unit 201, and generates image data that can be displayed on the display device 300. In addition, the image generating unit 203 superimposes the correction information calculated by the correction information calculation unit 202 on the image data. The image data on which the correction information has been superimposed is transmitted to the LED wall 300.

[0023] FIG. 2 shows an example of the hardware configuration of an image processing device 200 according to an embodiment.

[0024] The image processing device 200 includes a CPU 401, a RAM 402, a ROM 403, an auxiliary storage interface 404, an HDD 405, an input interface 406, an output interface 407, and a network interface 412. The components of the image processing device 200 are connected to one another by a system bus 408. The image processing device 200 is also connected to an external storage device 409 and an input device 411 via the input interface 406. The image processing device 200 is also connected to a monitor 410 via the output interface 407.

[0025] The CPU 401 executes a program stored in the ROM 403 using the RAM 402 as a work memory, and controls each component of the image processing device 200 via a system bus 408. This executes various processes described below. In addition to the CPU 401 and the RAM 402, a GPU for performing CG rendering for video display on the LED wall 300 and a VRAM for storing CG data (hereinafter, CG scene data) may be included. The HDD 405 is a storage device that stores various data handled by the image processing device 200. The CPU 401 writes data to the HDD 405 and reads data stored in the HDD 405 via the system bus 408. In addition to the HDD, various storage devices such as an optical disk drive and a flash memory may be used for the HDD 405.

[0026] The input interface 406 is, for example, a serial bus interface such as USB or IEEE1394. The image processing device 200 inputs data, commands, and the like from an external device via the input interface 406. The image processing device 200 of this embodiment acquires data from an external storage device 409 (e.g., a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory) via the input interface 406. The image processing device 200 of this embodiment also acquires user instructions input to an input device 411 via the input interface 406. The input device 411 is an input device such as a mouse or a keyboard, and inputs user instructions.

[0027] The output interface 407 is a serial bus interface such as USB or IEEE1394, similar to the input interface 406. The output interface 407 may be a video output terminal such as DVI or HDMI (registered trademark). The image processing device 200 outputs data, etc. to an external storage device via the output interface 407. The image processing device 200 of this embodiment outputs data processed by the CPU 401 (for example, real-time arrangement status of the camera and display device, etc.) to a monitor 410 (various image display devices such as a liquid crystal display) via the output interface 407.

[0028] The network interface 412 is an interface for connecting to a network such as Ethernet. In the embodiment, the imaging device 100, the display device 300, and the lighting device 400 can communicate with each other via the network interface 412. Note that the imaging device 100, the image processing device 200, the display device 300, and the lighting device 400 only need to be able to communicate with each other, so the connection form is not limited to an Ethernet interface, and may be USB or the like, and the type of communication interface does not matter. For example, the image processing device 200 and the lighting device 400 may be connected via a different interface, and these devices may be connected by combining multiple types of interfaces. Furthermore, the communication may be wired or wireless.

[0029] Furthermore, the image processing device 200 receives shooting information from the imaging device 100 via the network interface 412 and stores it in the RAM 402 via the system bus 408 .

[0030] The components of the image processing device 200 are not limited to those described above, and since they are not the main focus of this embodiment, a description thereof will be omitted. The image processing device 200 can be configured by a device such as a personal computer (PC) or a workstation. In this case, each processing unit implemented in the image processing device 200 is realized by a CPU executing a program operating in the PC described above.

[0031] The CPU 401 of the image processing device 200 receives the shooting information input from the imaging device 100, and thereby acquires the position information, line of sight information, and focal length information of the imaging device 100. Then, the CPU 200 generates information for correcting the image to be displayed on the LED wall 300 based on each piece of acquired information, causes the image generating unit 203 to generate the image to be displayed on the LED wall 300, and causes the generated image to be transmitted to the display device 300. These processes are performed at a timing controlled by a synchronization signal (not shown). Changes in the position and line of sight of the imaging device 100 are acquired at the interval of the synchronization signal, and correction information is generated by the image processing device, which is then superimposed on the image to be displayed on the display device 300 and output.

[0032] 3 is a further detailed functional block diagram of the image processing device 200 in this embodiment. It should be understood that the display device information storage unit 201 is realized, for example, by the HDD 405 in FIG. 2, and that the components other than the display device information storage unit 201 (such as the correction area calculation unit 222) are realized by the CPU 401 in FIG. 2 executing a program. The display device information storage unit 201 stores CG scene data and display position information of the display device 300. The display position information is also associated with the CG scene data and indicates the coordinate position on the CG scene at which the display device 300 is located. The position information input unit 220 receives shooting information (such as the position and orientation and focal length of the imaging device 100) from the transmitting / receiving device 103 of the imaging device 100 via the network interface 412.

[0033] The imaging information input unit 221 receives, via the network interface 412, the position and orientation information, focal length information, and sensor size information of the imaging device 100 contained in the imaging information received from the imaging device 100.

[0034] The correction area calculation unit 222 calculates a display area to be rendered on the display device 300 from the position information and line of sight information of the imaging device 100 input from the position information input unit 220, and the focal length information of the imaging device 100 input from the shooting information input unit 221. This becomes the correction area in which the image to be displayed should be corrected.

[0035] The pixel correction amount calculation unit 223 calculates the amount of correction for each LED pixel value on the LED panel constituting the display device 300. To calculate this correction amount, the pixel correction amount calculation unit 223 uses the position information and line of sight information of the imaging device 100 input from the position information input unit 220 and the shooting information input unit 221, the correction area on the display device 300 calculated by the correction area calculation unit 222, and the variation correction information for each LED panel of the display device information 210. The correction amount for each LED pixel value calculated here is output to the video generation unit 203 as correction information.

[0036] When the result of calculation by the pixel correction amount calculation unit 223 indicates that exposure correction is to be performed on the imaging device 100, the information transmission unit 204 transmits exposure correction information related to the exposure correction to the imaging device 100. In addition, the information transmission unit 204 transmits a control signal for performing illumination control on the illumination device 400.

[0037] The control unit 301 of the display device 300 separates the video signal sent from the video generation unit 203 into video data to be displayed on each LED panel, and transmits the data to each LED panel to display the video. For example, as in the embodiment, if the display device 300 is configured with two LED panels arranged side by side, the left half of the video signal is transmitted to the LED panel 302, and the right half is transmitted to the LED panel 303. At this time, it is assumed that the entire display device 300 can display a video with A pixels in the vertical direction and B pixels in the horizontal direction, and the two LED panels are the same size. Then, the LED panel 302 is in charge of displaying the pixel area of ​​the video from (0,0) to (B / 2-1,A-1), and the LED panel 303 is in charge of displaying the pixel area of ​​the video from (B / 2,0) to (B-1,A-1). In this way, when the arrangement of the LED panels is determined, it is determined which LED panel and at which position a certain pixel in the video to be displayed is to be reproduced. The LED panel 302 (303 is similar) is composed of a plurality of display elements (LED elements). For example, these LED elements are made up of elements of three color components, red (R), green (G), and blue (B), which are grouped together to form one pixel. Then, based on a video signal from the video generator 203 that includes the R, G, and B color information of each pixel, the LED panel 301 or the LED panel 302 emits light for the color component at a specified position, thereby displaying an image on the display screen composed of the LED panels 301 and 302.

[0038] [Processing flow of the entire image processing device] Next, the overall processing flow of the image processing device 200 in this embodiment will be described with reference to the flowchart in FIG.

[0039] In S501, the correction area calculation unit 222 acquires, from the image capturing device 100, position information (including the attitude) and line-of-sight information (information indicating the optical axis direction) of the image capturing device 100.

[0040] In S502 , the correction area calculation unit 222 acquires focal length information from the image capture device 100 .

[0041] In S503 , the correction area calculation unit 222 acquires the layout information of the display device 300 from the display device information storage unit 201 .

[0042] In S504, the correction area calculation unit 222 calculates a correction area on the display device 300 that is an area that falls within the shooting angle of view of the image capture device 100, based on the information acquired in S501 to S503. A specific example will be described with reference to FIG. 5. FIG. 5 is a diagram of the display device 300 viewed from the front, showing how the image capture device 100 captures an image from an oblique direction. As shown in FIG. 5, the correction area calculation unit 222 determines four vectors 603A to 603D that extend from the imaging lens of the image capture device 100 to the four corners of the angle of view, based on the focal length information, sensor size information, position and orientation 602 (hence line of sight 601), and line of sight direction 601 of the image capture device 100. Then, the correction area calculation unit 222 obtains the intersection positions between the display surface of the display device 300 and each of the four vectors 603A to 603D, and determines the dashed line area surrounded by the four intersection positions as the correction area 606. In addition, the correction area 606 may be expanded by a predetermined ratio in the vertical and horizontal directions so as to be able to handle sudden movements of the imaging device. The correction area calculation unit 222 outputs information representing the correction area 606 obtained by the above calculation to the pixel correction amount calculation unit 223.

[0043] In S505, the pixel correction amount calculation unit 223 calculates the correction amount of each pixel in the correction area 606 on the display device 300 based on information representing the correction area 606 calculated by the correction area calculation unit 222 and the position information and line of sight information of the imaging device input from the position information input 220 (described in detail later).

[0044] In S506, the pixel correction amount calculation unit 223 calculates the correction amount of the display device 300 and the exposure correction amount of the imaging device 100 based on the correction amount of each pixel calculated in S505. Then, the pixel correction amount calculation unit 223 outputs the calculated correction amount of the display device 300 to the image generation unit 203 as pixel correction amount information. In addition, the pixel correction amount calculation unit 223 outputs the calculated exposure correction amount of the imaging device 100 to the information transmission unit 204 as exposure amount correction information. In addition, the pixel correction amount calculation unit 223 also stores the output correction amount and exposure correction amount in the RAM 402. Note that details of the calculation process by the pixel correction amount calculation unit 223 will be described later.

[0045] In S507, the video generation unit 203 renders a CG scene to be displayed on the display device 300 based on the CG scene data previously stored in the display device information storage unit 201. The CG scene may only be within the correction area 606 on the display device 300 output from the pixel correction amount calculation unit 223. Therefore, the video generation unit 203 performs rendering only on the portion corresponding to the correction area 606. Then, based on the pixel correction amount information calculated by the pixel correction amount calculation unit 223 through S505 and S506, the video generation unit 203 calculates the amount of correction of pixel values ​​for each pixel within the correction area (described in detail later).

[0046] Then, in S508, the image generating unit 203 transmits image data including pixel value data within the correction area after correction to the display device 300 to display the image. At this time, the image generating unit 203 transmits only pixel value information within the correction area, so that for pixels other than the correction area, a method is adopted in which a different image is displayed or nothing is displayed. In some cases, for pixels outside the correction area, the same data as the previous time may be output.

[0047] In S509, the pixel correction amount calculation unit 223 determines whether or not the exposure amount correction calculated in S506 has been generated for the imaging device 100. If it has been generated, the pixel correction amount calculation unit 223 advances the process to S510, and if it has not been generated, the pixel correction amount calculation unit 223 ends this process.

[0048] In S510, the information transmission unit 204 adjusts the brightness of the lighting device 400 that illuminates the subject (performer) positioned in front of the display device 300, based on the exposure amount correction information calculated by the pixel correction amount calculation unit 223 and stored in the RAM 402. For example, as shown in Fig. 13, the relationship between the exposure correction amount of the image capture device 100 and the brightness of the lighting is stored as a table, and a value indicating the brightness after adjustment is determined based on the exposure amount correction information and the current brightness information, and the value is output to the lighting device 400. For example, assuming that the lighting device 400 is operating with an output of 50, and the exposure amount correction information is +1 / 3 step, the lighting output value is reduced by "5" to an output of "45".

[0049] In S511, the information transmission unit 204 corrects the exposure of the imaging device 100 based on the exposure amount correction information calculated in S506. For example, if the exposure amount correction information is -1 / 3 step, the information transmission unit 204 transmits information to the imaging device 100 for shortening the shutter speed by 1 / 3 step. Note that in this description, the shutter speed is used for exposure amount correction, but other shooting conditions (aperture value, ISO sensitivity, etc.) may be changed, or multiple items may be changed. The entire processing flow of the image processing device 200 in this embodiment has been described above.

[0050] Next, details of the process of the pixel correction amount calculation unit 223 in S505 of FIG. 4 will be described with reference to the flowchart of FIG.

[0051] In S701, the pixel correction amount calculation unit 223 acquires the position information and line of sight information input from the imaging device 100.

[0052] In S702 , the pixel correction amount calculation unit 223 acquires the correction area information calculated by the correction area calculation unit 222 .

[0053] In S703, the pixel correction amount calculation unit 223 calculates the degree between the normal vector 803 and the line of sight from the imaging device 100 for each pixel in the correction area on the display device 300, based on the position information and line of sight information of the imaging device 100 acquired in S701, and the correction area information acquired in S702.

[0054] Specifically, the pixel correction amount calculation unit 223 calculates which specific pixel position corresponds to the correction area calculated in S504, as shown in FIG. 7. Here, reference numeral 801 indicates an LED element at coordinates (x, y) in the display device 300. The LED element 801 corresponds to one pixel of an image displayed by the display device 300. For example, if the correction area 606 is an area indicated by a thick frame in FIG. 7 (shown as a rectangle for easy understanding), the area surrounded by the correction area 606 is an area with coordinates (x+3, y+1) at the upper left corner and coordinates (x+5, y+3) at the lower right corner. At this time, the pixel correction amount calculation unit 223 calculates an angle 804 between a vector 802 connecting the position 602 of the imaging device 100 and the center of the LED element (x+3, y+1) and the plane of the correction area 606. Similarly, the pixel correction amount calculation unit 223 obtains an angle between the vector connecting the position of the imaging device 100 and the center of the LED element and the correction area 606 for each pixel up to the coordinate (x+5, y+3). In order to know the direction from which the LED element is photographed, as shown in FIG. 12, an x-axis 1501 that intersects with the normal vector 803 at a right angle, is parallel to the horizontal direction of the display device 300, and has a positive rightward direction, and a y-axis 1502 that is parallel to the vertical direction of the display device 300 and has a positive upward direction are defined. At this time, the normal vector 803 is defined as the z-axis. Next, the pixel correction amount calculation unit 223 breaks down the vector 802 into three parts: a movement amount 802X in the x-axis direction, a movement amount 802Y in the y-axis direction, and a movement amount 802Z in the z-axis direction. Then, the pixel correction amount calculation unit 223 calculates two vectors: an x-axis vector 802XZ centered on the normal vector 803, and a y-axis vector 802YZ. The angle θ between these vectors 802XZ and the normal vector 803 XZ , and the angle θ between vector 802YZ and normal vector 803 YZ Find the angle between these twoXZ , θ YZ This is the angle between the two.

[0055] In this case, if the normal vector 803 is a = (a1, a2, a3) and the vector 802 is b = (b1, b2, b3), then the vector 803XZ is a XZ =(a1,0,a3), vector 802YZ is a YZ = (0, a2, a3). And the angle θ between the vector 802XZ and the normal vector 803 XZ , and the angle θ between vector 802YZ and normal vector 803 YZ can be calculated using the following equations (1) and (2).

[0056]

number

[0057] In S704, the pixel correction amount calculation unit 223 calculates the correction amount for each pixel based on the angle of each pixel calculated in S703. Specifically, the correction amount for the LED element may be determined by referring to a displacement characteristic information table as shown in FIG. 9, which corresponds to the angle. The displacement characteristic information is a table in which the angle is determined depending on whether the imaging device 100 captures the image from the left or right of the display device 300, with the angle being set to 0 when the display device 300 and the imaging device 100 face each other, and the correction amount for the pixel value corresponding to the angle is shown. In FIG. 9, the LED element is displayed with the same facing direction as the normal vector of the display surface of the display device as 0 degrees, the leftward direction with respect to the x-axis direction (described later) as plus, and the rightward direction as minus. FIG. 8 shows only the displacement characteristic information with respect to the x-axis direction, but in reality, the line-of-sight vector from the camera to the LED element needs to be calculated as an angle within a hemispherical range with the LED element as the center, as shown in FIG. 11. For example, in FIG. 11, as in FIG. 8, when the x-axis 1501, y-axis 1502, and normal vector 803 are the z-axis, the position where the display device (LED wall) faces directly, that is, the position where it is 0 degrees, is point 1503. The points indicating the correction amount can be set with the left side being negative and the right side being positive, centered on normal vector 803 which is the z-axis. For example, a point at minus 30 degrees in the x-axis direction is a point 30 degrees to the left of normal vector 803, and is therefore at point 1504. The angle deformation characteristic information shown in FIG. 9 corresponds to an angle on a plane formed by the x-axis and z-axis, such as point 1504 in FIG. 11. Since the LED element has angle dependency not only in the x-axis direction but also in the y-axis direction, it is possible to generate angle deformation characteristic information corresponding to the y-axis direction by rotating the angle deformation characteristic information shown in FIG. 9 by, for example, 180 degrees counterclockwise. At this time, since the correction coefficients are different between the left-facing and right-facing directions, it is advisable to actually perform an interpolation calculation between the negative angle value and the positive angle value when rotating 180 degrees counterclockwise. For example, the average value of the correction amounts between the information on the point at minus 80 degrees and the point at plus 80 degrees corresponds to point 1506, which is minus 80 degrees on the plane formed by the y-axis and z-axis. In this case, the downward direction with respect to the y-axis is negative and the upward direction is positive. In this way, the correction amount can be interpolated and defined in a hemisphere with the normal vector 803 as the center for the correction area 606.

[0058] Next, in the example shown in FIG. 7, the angle 804 is θ XZ Plus 45 degrees, θ YZ If the angle is 0 degrees, the light emitted from the LED element will be directed leftward when the camera position information and the position of the LED element are referred to, so the +45 degree part in FIG. 9 is referred to. Also, since each LED element is composed of three types of pixels, red (R), green (G), and blue (B), one correction amount is stored for each color information of R, G, and B. Note that in FIG. 9, the angles are 30 degrees, 45 degrees, 60 degrees, and 80 degrees, so for example, angles greater than 80 degrees are equivalent to 80 degrees, and angles between 80 degrees and 60 degrees can be calculated by linearly interpolating the data of 80 degrees and 60 degrees. Also, the correction amount information by angle may be in smaller increments (such as every other degree) than in FIG. 8, and if the display device 300 has LED elements of other colors, such as white LEDs (W) in addition to R, G, and B, correction amounts for other colors may also be stored. For example, when a white LED (W) is provided in addition to R, G, and B, a W element correction amount may be provided in addition to the correction amounts for the R element, G element, and B element.

[0059] In S705, the pixel correction amount calculation unit 223 corrects the effect of individual differences of LED panels for the correction amount of each pixel calculated in S704. Due to factors such as variations in the accuracy of parts during manufacturing, LED panels may output different brightnesses even when the same video signal is applied. For this reason, three types of correction coefficients, red (R), green (G), and blue (B), are provided for each panel as individual difference information. Specifically, as shown in FIG. 10, each panel position has R, G, and B correction coefficients (ratios) relative to a preset reference value of brightness. If the display position of a pixel (x, y) in the image is known, the corresponding panel can be identified. In other words, the value of the correction coefficient can be obtained from the pixel position.

[0060] In S706, the pixel correction amount calculation unit 223 outputs the correction amount data for each pixel calculated up to S705 to the image generation unit 203.

[0061] Next, details of the process of the pixel correction amount calculation unit 223 in S506 in FIG. 4 will be described with reference to the flowchart in FIG.

[0062] In S801, the pixel correction amount calculation unit 223 returns the exposure correction amount of the imaging device 300 to an initial value.

[0063] In S802, the pixel correction amount calculation unit 223 receives the correction amount data for each pixel calculated in S505. Additionally, the pixel correction amount calculation unit 223 receives the pixel values ​​in the correction area rendered by the video generation unit 203.

[0064] In S803, the pixel correction amount calculation unit 223 calculates the pixel value of each pixel to be output to the display device 300 from the correction amount data of each pixel input in S802 and each pixel value in the correction area. Specifically, the pixel correction amount calculation unit 223 multiplies the pixel value in the correction area by the correction amount data of each pixel, and sets the value obtained as the pixel value after correction.

[0065] In S804, the pixel correction amount calculation unit 223 refers to the maximum value among the calculated pixel values ​​of each pixel, and determines whether or not the maximum pixel value controllable by the display device 300 is exceeded.

[0066] For example, suppose that the display device 300 controls pixel values ​​in 8 bits. Suppose that the value of a certain pixel in the correction area rendered by the image generation unit 203 is "217" and the correction amount of that pixel is "1.20". In this case, the pixel value after correction is "260" (=217×1.20), which exceeds the maximum 8-bit value "255" that the display device 300 can input. Therefore, in this example, in S804, the pixel correction amount calculation unit 223 determines that the pixel value after correction exceeds the maximum controllable pixel value.

[0067] In S804, if the pixel correction amount calculation unit 223 determines that the correction amount is exceeded, the process proceeds to S805, and if it determines that the correction amount is not exceeded, the process proceeds to S808.

[0068] In S805, the pixel correction amount calculation unit 223 calculates the exposure correction amount of the imaging device 100. Specifically, when the pixel value after correction exceeds the maximum value of 8 bits that can be controlled by the display device 300, the pixel correction amount calculation unit 223 reduces the pixel values ​​of the display device 300 as a whole. Then, the pixel correction amount calculation unit 223 compensates for the reduced amount with the exposure of the imaging device 100 so as to maintain the overall brightness of the entire image. For example, the pixel correction amount calculation unit 223 multiplies the brightness of the entire image by 0.794 and increases the exposure of the imaging device 100 by 1 / 3 step accordingly so as to maintain the overall brightness. For example, when the pixel value after correction at a certain pixel position of the display device 300 is "260", the pixel correction amount calculation unit 223 multiplies all pixel values ​​of the image to be displayed on the display device 300 (image in the correction area) by 0.794 (hereinafter, the image after all pixel values ​​in the correction area are multiplied by 0.794 is referred to as "pixel correction information"). As a result, the previous pixel value becomes "206." Note that even if multiplied by 0.794, it is possible that the maximum value of 8 bits will be exceeded. In this case, it is further multiplied by 0.794, and the exposure of the image capture device 100 is increased by another 1 / 3 stop to check whether the maximum value is exceeded. In this way, the pixel correction amount calculation unit 223 performs correction calculation again until the value becomes equal to or less than the maximum value of 8 bits, and increases the exposure correction amount of the image capture device 100 each time.

[0069] In S806, the pixel correction amount calculation unit 223 stores the exposure correction amount of the imaging device 100 obtained by the calculation in S805 in the RAM 402, and outputs it to the information transmission unit 204. The information transmission unit 204 transmits the exposure correction amount input from the pixel correction amount calculation unit 223 to the imaging device 100. The imaging device 100 corrects the exposure according to the exposure correction amount and performs imaging.

[0070] In S807, the pixel correction amount calculation unit 223 updates each pixel value in the correction area obtained in S803 with the pixel correction amount information calculated in S805.

[0071] In S808, the pixel correction amount calculation unit 223 outputs the pixel correction amount information to the image generation unit 203. If the determination in S804 is No, this pixel correction amount information is image information having the pixel values ​​calculated in S803, and the initial exposure amount is maintained for the imaging device 100. On the other hand, if the determination in S804 is Yes, the pixel correction amount information is image information having pixel values ​​obtained by the update process in S807.

[0072] The image generating unit 203 renders an image to be displayed on the display surface of the display device 300 (two LED panels in this embodiment) based on the CG scene data stored in the display device information 210 and the arrangement information of the display device 300. At this time, the display position of the CG scene is determined from the arrangement information of the display device 300 associated with the CG scene data. Then, the image generating unit 203 multiplies the correction amount of each pixel calculated by the correction amount calculating unit 202 by the rendered image, and transmits the resulting image data to the display device 300. For example, if the pixel value of a certain pixel (a, b) in the rendering result is (R, G, B) = (100, 100, 150), the correction amount is (1.12, 1.11, 1.19), and the panel position is (1, 1), the pixel value after correction is (R', G', B') = (113, 113, 181).

[0073] As described above, according to this embodiment, regardless of the relative positional relationship between the imaging device 100 and the display device 300 or the brightness variations among the LED panels that make up the display device 300, the image captured by the camera can be displayed as an image on the LED wall with a constant brightness.

[0074] In the above embodiment, the individual differences of the LED panels 302 and 303 constituting the display device 300 are held in advance. However, a method of parameterizing the influence of the individual differences by measurement before using the display device 300 may be adopted. For example, an individual difference information generating unit (not shown) is provided in the image processing device 200, and the individual difference information generating unit generates individual difference information based on the video data of the LED panels constituting the display device 300 obtained by the imaging device 100. Specifically, a test chart prepared in advance is displayed on the display device 300, and the image is captured by the imaging device 100. At this time, the ratio of the average values ​​of the pixels of the chart portion of each LED panel may be held. At this time, the LED panels 302 and 303 constituting the display device 300 are displayed one by one, and the imaging device 100 is placed facing each LED panel at a position that is the center of the angle of view, and the image is captured, allowing accurate measurement. In addition, the test chart may be a monochromatic chart using signal values ​​that make the LED panels uniform in color, such as gray, red, green, and blue.

[0075] In the above embodiment, the viewing range of the imaging device 100 within the display screen of the display device 300 is defined as a correction area, and pixel values ​​are corrected for the correction area. However, if the display screen of the image processing device 200 is sufficiently large with respect to the maximum angle of view of the imaging device 100 and the image processing device has sufficient computing power, the image processing device 200 does not require information regarding the angle of view (focal length) of the imaging device 100, and may correct all display pixels on the display screen of the display device 300 from information indicating its position and orientation (position and optical axis direction).

[0076] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0077] The disclosure of this specification includes the following image processing device, method, and program. (Item 1) An image processing device that generates an image to be displayed on a display device having one or more display panels arranged as a display screen, the display panels being made up of display elements having brightness characteristics that depend on an angle, comprising: an acquisition means for acquiring information regarding a position and an attitude of an imaging device for capturing an image of a subject against a background of an image displayed on a display screen of the display device; a calculation means for calculating a correction coefficient for each display element based on the display position of the display device and the information acquired by the acquisition means; a correction means for correcting each pixel in the image data to be displayed on the display device by the correction coefficient calculated by the calculation means, 2. An image processing apparatus comprising: a display device that displays image data corrected by said correction means; (Item 2) The image processing device described in item 1, characterized in that the calculation means determines a vector from each of the display elements to the imaging device based on the display position of the display device and the information acquired by the acquisition means, and calculates a correction coefficient for each display element that depends on the characteristics. (Item 3) The image processing device further includes a control unit that controls the correction unit, outputs the corrected image data to the display device, and generates and outputs correction information related to exposure for the image capture device, The control means a determination unit that determines whether or not the image data corrected by the correction unit contains pixel data that exceeds a maximum value that can be input to the display device, The control means When the determination result of the determination means indicates that the corrected image data does not include pixel data exceeding a maximum value that can be input to the display device, the corrected image data is output to the display device, and an initial exposure amount is maintained for the imaging device; When the determination result of the determination means indicates that the corrected image data includes pixel data exceeding the maximum value that can be input by the display device, the image data obtained by performing a second correction on the corrected image data so that the pixel data is equal to or less than the maximum value is output to the display device, and correction information for increasing the amount of exposure based on the second correction is generated and output to the imaging device. 3. The image processing device according to item 1 or 2. (Item 4) 4. The image processing device according to item 3, wherein the control means further generates information indicating an illumination intensity for an external lighting device and outputs the information to the lighting device. (Item 5) the acquiring means further acquires information relating to an angle of view from the imaging device, the calculation means includes a determination means for determining, as a correction area, an area within the angle of view on the display screen based on information relating to a position of the display device, a position and an attitude of the imaging device, and the angle of view; The calculation means calculates a correction coefficient within the correction area. 5. The image processing device according to any one of items 1 to 4. (Item 6) The display device further includes a means for acquiring information representing individual differences in light emission of each of the display panels constituting the display screen of the display device, The correcting means further corrects the corrected image data based on information representing the individual difference for each display panel. 6. The image processing device according to item 5, (Item 7) The information representing the individual differences is information indicating the ratio of brightness to the reference values ​​of R, G, and B of each display panel. 7. The image processing device according to item 6, (Item 8) The correction means generates and corrects image data to be displayed on the display device by performing rendering based on the CG data stored in the storage means. 8. The image processing device according to any one of items 1 to 7, (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the display element is an LED element, and the display panel is an LED panel. (Item 10) A method for controlling an image processing device that generates an image to be displayed on a display device having a display screen in which one or more display panels each composed of a display element having an angle-dependent brightness characteristic are arranged, comprising the steps of: an acquisition step of acquiring information regarding a position and an attitude of an imaging device for capturing an image of a subject against a background of an image displayed on a display screen of the display device; a calculation step of calculating a correction coefficient for each display element based on the display position of the display device and the information acquired in the acquisition step; a correction step of correcting each pixel in the image data to be displayed on the display device by the correction coefficient calculated in the calculation step, The image processing apparatus control method further comprises outputting the image data corrected in the correction step to the display device. (Item 11) A computer program that, when read and executed by a computer, causes the computer to function as each of the means of the device described in any one of items 1 to 9.

[0078] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 100: Imaging device, 200: Image processing device, 300: Display device, 201: Display device information storage unit, 202: Correction information calculation unit, 203: Video generation unit, 204: Information transmission unit, 220: Position information input unit, 221: Shooting information input unit, 222: Correction area calculation unit

Claims

1. an acquisition means for acquiring information about the position and orientation of an imaging device for capturing an image of a subject with an image displayed on a display device as a background; a calculation means for calculating a correction coefficient corresponding to each display element of the display device based on the display position of the display device and the information acquired by the acquisition means; and a correction unit that corrects the pixel value of each pixel in the image displayed on the display device with the correction coefficient calculated by the calculation unit.

2. The image processing device described in Claim 1, characterized in that the display device is an image processing device in which one or more display panels composed of display elements having angle-dependent brightness characteristics are arranged to form a display screen.

3. The image processing device according to claim 2, characterized in that the calculation means determines a vector directed from each of the display elements to the imaging device based on the display position of the display device and the information acquired by the acquisition means, and calculates a correction coefficient that depends on the characteristics of each display element.

4. The image capturing device further includes a control unit that controls the correction unit, outputs the corrected image to the display device, and generates and outputs correction information related to exposure for the image capturing device, The control means a determination unit that determines whether or not the image corrected by the correction unit includes a pixel whose value exceeds the maximum value that can be input to the display device, The control means If the determination result of the determination means indicates that the corrected image does not include a pixel whose value exceeds the maximum value that can be input to the display device, outputting the corrected image to the display device and maintaining an initial exposure amount for the imaging device; If the determination result of the determination means indicates that the corrected image contains a pixel whose value exceeds the maximum value that can be input to the display device, the corrected image is corrected again so that the pixel value becomes equal to or less than the maximum value, and the resulting image is output to the display device, and correction information for increasing the exposure amount based on the correction again is generated and output to the imaging device.

2. The image processing device according to claim 1, wherein:

5. 5. The image processing apparatus according to claim 4, wherein the control means further generates information indicating the intensity of illumination for an external lighting device and outputs the information to the lighting device.

6. the acquisition means further acquires information relating to an angle of view from the imaging device, the calculation means includes a determination means that determines, based on information relating to a position of the display device, a position and orientation of the imaging device, and the angle of view, an area on the display screen that falls within the angle of view as a correction area; The calculation means calculates a correction coefficient within the correction area.

2. The image processing device according to claim 1, wherein:

7. The display device further includes a means for acquiring information representing individual differences in light emission of each display panel constituting a display screen of the display device, The correcting means further corrects the corrected image based on information representing the individual differences of each display panel.

7. The image processing device according to claim 6,

8. The information representing the individual differences is information indicating the ratio of brightness to the reference values ​​of R, G, and B of each display panel.

8. The image processing device according to claim 7,

9. The correction means generates and corrects an image to be displayed on the display device by performing rendering based on the CG data stored in the storage means.

2. The image processing device according to claim 1, wherein:

10. 3. The image processing apparatus according to claim 2, wherein the display element is an LED element, and the display panel is an LED panel.

11. The image processing device according to claim 1, further comprising an output means for outputting the image corrected by the correction means to the display device.

12. an acquisition step of acquiring information about the position and orientation of an imaging device for capturing an image of a subject with an image displayed on a display device as a background; a calculation step of calculating a correction coefficient corresponding to each display element of the display device based on the display position of the display device and the information acquired in the acquisition step; a correction step of correcting the pixel value of each pixel in the image displayed on the display device with the correction coefficient calculated in the calculation step.

13. A computer program that, when read and executed by a computer, causes the computer to execute each step of the method according to claim 12.

Images