Image processing device

Abstract

To easily analyze a virtual area and a real area.SOLUTION: An image processing device includes: receiving means for receiving image data; detection means for detecting a first image area corresponding to a virtual area and a second image area corresponding to a real area from the image data; first generation means for analyzing the first image area to generate first analyzed data; second generation means for analyzing the second image area to generate second analyzed data; and display means for displaying the first analyzed data and the second analyzed data together with the image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image analysis technique and an image processing device using the same technique. [Background technology]

[0002] Virtual production using in-camera VFX with LED walls is becoming increasingly common. An LED wall is a display device with a grid of LEDs. In in-camera VFX, pre-recorded footage is displayed on the LED wall, and performers and sets are positioned between the LED wall and the camera, which then records the LED wall, performers, and sets. The area of ​​the footage captured by in-camera VFX that captures the performers and sets is referred to as the real area, while the area that captures the LED wall is referred to as the virtual area.

[0003] Patent Document 1 discloses a technology for displaying a vectorscope for a partial region of an image. As a method for specifying the partial region, a method is disclosed in which the user specifies the region from each pixel of the image by color or brightness using a touch panel.

[0004] In virtual production, there is a demand for the ability to analyze the virtual and real areas separately using image analysis devices such as waveform monitors and vectorscopes. The virtual area is an image taken with one camera, displayed on an LED wall, and then shot again with a second camera, while the real area is an image taken directly with the second camera. This is because the image quality of the virtual and real areas can differ depending on the characteristics of the cameras and LED walls involved, and it is necessary to check this. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-16260 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the above-mentioned conventional example, although it is possible to display a vectorscope of a partial area, in order to analyze the virtual area and the real area separately, the user had to specify the area. Furthermore, it was necessary to specify the area every time the area moved in the video. Therefore, an object of the present invention is to easily analyze the virtual area and the real area separately. [Means for solving the problem]

[0007] The image processing device has a receiving means for receiving image data, a detecting means for detecting from the image data a first image area corresponding to a virtual area and a second image area corresponding to a real area, a first generating means for analyzing the first image area and generating first analytical data, a second generating means for analyzing the second image area and generating second analytical data, and a display means for displaying the first analytical data and the second analytical data together with the image data. [Effects of the Invention]

[0008] According to the present invention, the virtual area and the real area can be easily analyzed. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a display device according to first and second embodiments. [Figure 2] FIG. 10 is a diagram showing an example of video data according to the first and second embodiments. [Figure 3] FIG. 10 is a diagram showing an example of display of composite video data according to the first and second embodiments. [Figure 4] 10 is a flowchart showing a control flow according to a second embodiment. [Figure 5] FIG. 10 is a diagram showing an example of display of composite video data according to the second embodiment; [Figure 6]FIG. 10 is a block diagram showing a configuration example of a display device according to a third embodiment; [Figure 7] FIG. 10 is a diagram showing an example of first input moving image data and second input moving image data according to the third embodiment; [Figure 8] FIG. 13 is a diagram showing an example of video data according to the third embodiment; [Figure 9] FIG. 13 is a diagram showing an example of display of composite video data according to the third embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0010] First Embodiment Hereinafter, an embodiment of the present invention will be described. An example will be described in which a display device has an image processing device (hereinafter also referred to as an image analysis device) according to this embodiment built in. Note that the image analysis device according to this embodiment may be a device (such as a personal computer) separate from the display device.

[0011] Fig. 1 is a block diagram showing an example of the configuration of a display device according to this embodiment. The display device 100 in Fig. 1 includes a receiving unit 101, a detecting unit 102, a control unit 103, a memory unit 104, a combining unit 105, a display unit 106, a generating unit 107, and a generating unit 108.

[0012] The receiving unit 101 acquires video data (image data) and outputs the video data to the detecting unit 102, the synthesizing unit 105, the generating unit 107, and the generating unit 108. In this embodiment, the receiving unit 101 acquires frame data for each frame of the video from an external device. The receiving unit 101 then outputs the acquired frame data to each subsequent unit. The receiving unit 101 is an input terminal that complies with standards such as SDI and HDMI (registered trademark). The external device is an imaging device, a playback device, or the like.

[0013] The detection unit 102 acquires the video data output from the receiving unit 101, detects region information of the video data, and outputs the detected region information to the control unit 103, the generation unit 107, and the generation unit 108. Here, the region information is identification information that identifies whether each pixel is a real region (second image region) or a virtual region (first image region). In this embodiment, this region information is generated by an external device. Information indicating the real region and the virtual region is assigned based on, for example, a distance map (distance information) obtained by a depth sensor, with pixels farther away than a predetermined position being designated as virtual regions and the rest being designated as real regions. In addition, the region information is superimposed on the video data as metadata. Examples of the metadata include ANC data standardized in SDI and InfoFrame standardized in HDMI (registered trademark).

[0014] The control unit 103 controls the processing of each unit of the display device 100. The control unit 103 is connected to each unit via a control bus, which is not shown in the figure due to its complexity. For example, the control unit 103 is an arithmetic processing circuit that executes a program stored in the memory unit 104 to control the processing of each block of the display device 100. In this embodiment, the control unit 103 controls the synthesis unit 105 based on the area information output from the detection unit 102.

[0015] The memory unit 104 stores programs, parameters, etc. The programs and parameters stored in the memory unit 104 are read and written by each block of the display device 100.

[0016] The synthesis unit 105 synthesizes the moving image data with the first waveform monitor output by the generation unit 107 and the second waveform monitor output by the generation unit 108 based on the control content output from the control unit 105, thereby generating synthesized moving image data. Then, the synthesis unit 105 outputs the synthesized moving image data to the display unit 106.

[0017] Display unit 106 displays on a display surface a moving image based on the composite moving image data output from composition unit 105. Display unit 106 is, for example, a liquid crystal display unit having a liquid crystal panel and a backlight unit, or an organic EL display panel.

[0018] The generating unit 107 acquires the video data output from the receiving unit 101 and the region information output from the detecting unit 102, and generates a waveform monitor (first waveform monitor) for pixels in the video data that are designated as virtual regions by the region information. More specifically, the generating unit 107 analyzes one frame of data (image data) from the video data and identifies the pixels designated as virtual regions. The generating unit 107 then generates the analysis results (first analysis data) as the first waveform monitor. Here, the waveform monitor displays, for example, the signal level of the image relative to the horizontal resolution of the video. The signal level is, for example, a value corresponding to one of Y, Cb, Cr, R, G, and B. The waveform monitor may also display multiple signal levels (for example, four: Y, R, G, and B) in parallel.

[0019] The generation unit 108 acquires the video data output from the receiving unit 101 and the region information output from the detection unit 102, and generates a waveform monitor (second waveform monitor) for pixels in the video data that are designated as real regions by the region information. More specifically, the generation unit 108 analyzes one frame's worth of data (image data) from the video data and identifies the pixels designated as real regions. The generation unit 108 then generates the analysis results (second analysis data) as the second waveform monitor. Here, the waveform monitor displays, for example, the signal level of the image relative to the horizontal resolution of the video. The signal level is, for example, a value corresponding to Y, Cb, Cr, R, G, and B. The waveform monitor may also display multiple signal levels (for example, four: Y, R, G, and B) in parallel.

[0020] Having explained the operation of each unit, we will now explain the procedure for integrating those operations to display the first waveform monitor and second waveform monitor on display unit 106. When receiving unit 101 receives video data, detection unit 102 extracts region information from the video data and outputs it to control unit 103 and generation units 107 and 108. Generation units 107 and 108 refer to the region information and generate the first waveform monitor and second waveform monitor. Synthesis unit 105 synthesizes the first waveform monitor and second waveform monitor into video data and outputs it to display unit 106 as synthesized video data. Display unit 106 displays the synthesized video data including the first waveform monitor and second waveform monitor.

[0021] FIG. 2 is a diagram showing an example of video data. Wall 201 and window frame 202 are an art set. A performer 203 is positioned in front of wall 201 of the art set. An outside scene 211 and a standing tree 212 are positioned inside window frame 202. The outside scene 211 and the standing tree 212 are photographed in advance and displayed on an LED wall. In this state, photographing wall 201, window frame 202, performer 203, and LED wall results in the video data shown in FIG. 2. Wall 201, window frame 202, and performer 203 are real areas. The outside scene 211 and the standing tree 212 are virtual areas.

[0022] FIG. 3 is a diagram showing an example of the display of composite video data. Waveform monitor 301 is a first waveform monitor generated by generation unit 107, which visualizes the characteristics by analyzing the virtual region of the video data shown in FIG. 2. Waveform monitor 302 is a second waveform monitor generated by generation unit 108, which visualizes the characteristics by analyzing the real region of the video data shown in FIG. 2. Items that are the same as in FIG. 2 are denoted by the same reference numerals as in FIG. 2. Waveform monitors 301 and 302 are synthesized with the video data shown in FIG. 2 to produce the composite video data shown in FIG. 3.

[0023] This embodiment eliminates the need for the user to specify the areas when analyzing the virtual area and the real area separately. By referring to the analysis results of the first and second waveform monitors, the user can easily check the brightness distribution of the virtual and real areas and adjust the camera exposure and the brightness of the LED wall. The user can also adjust the image quality of the virtual area by changing the brightness and white balance of the LED wall while referring to the analysis results of the virtual area. Furthermore, the user can adjust the image quality of the virtual and real areas to be equal or intentionally different by changing the brightness and white balance of the LED wall while comparing the analysis results of the virtual and real areas.

[0024] In this embodiment, an example has been described in which separate waveform monitors are displayed for the virtual area and the real area, but instead of the waveform monitors, it is also possible to use a video analysis means including a vectorscope.

[0025] <Second embodiment> In the first embodiment, an example in which a first waveform monitor and a second waveform monitor are displayed is described. In the second embodiment, an example in which the display of the first waveform monitor and the second waveform monitor, or the display of only the first waveform monitor, is switched depending on the user's setting operation and the presence or absence of area information is described. The following mainly focuses on the differences from the first embodiment.

[0026] The display device 100 has additional means for the user to perform setting operations, such as operation keys and an OSD menu. The display device 100 has three setting values ​​for the waveform monitor display setting: "Off," "Auto," and "On," which the user can select.

[0027] FIG. 4 is a flowchart showing the control flow of the display device 100, which is executed by the control unit 103 every time the receiving unit 101 acquires one frame of video data.

[0028] In S401, if the waveform monitor display setting is "Off", the control unit 103 determines to execute the process of S404. If the waveform monitor display setting is not "Off", that is, if it is "Auto" or "On", the control unit 103 determines to execute the process of S402.

[0029] In S402, if the waveform monitor display setting is "Auto", the control unit 103 determines to execute the process of S403. If the waveform monitor display setting is not "Auto", that is, if it is "On", the control unit 103 determines to execute the process of S405.

[0030] To summarize the operations of S401 and S402, when the waveform monitor display setting is "Off", the process proceeds to S404; when the waveform monitor display setting is "Auto", the process proceeds to S403; and when the waveform monitor display setting is "On", the process proceeds to S405.

[0031] In S403, if the detection unit 102 detects region information from the video data, the control unit 103 determines to execute the process of S405. If the detection unit 102 does not detect region information from the video data, the control unit 103 determines to execute the process of S404. Note that since region information is superimposed on video data as ANC data in SDI or InfoFrame in HDMI (registered trademark), it cannot be detected if the external device does not superimpose it.

[0032] In S404, the display unit 106 displays one waveform monitor of the entire region of the video data. At this time, the generation unit 107 generates the waveform monitor of the entire region of the video data acquired from the receiving unit 101 as a first waveform monitor. The control unit 103 combines the first waveform monitor with the video data and outputs the combined video data to the display unit 106. The display unit 106 displays the combined video data including the first waveform monitor.

[0033] 5 is a diagram showing an example of the display of composite video data when one waveform monitor of the entire region of the video data is displayed. Waveform monitor 501 is a first waveform monitor generated by generation unit 107, which analyzes the entire region of the video data and visualizes its characteristics. Items that are the same as those in FIG. 2 are indicated by the same reference numerals as in FIG. 2. Waveform monitor 501 is synthesized with the video data shown in FIG. 2 to produce the composite video data shown in FIG. 5.

[0034] In S405, the display unit 106 displays a first waveform monitor corresponding to the virtual area of ​​the video data and a second waveform monitor corresponding to the real area. Since this is the same as in the first embodiment, details are omitted.

[0035] In this embodiment, an example has been described in which, when region information is detected from video data, a first waveform monitor corresponding to a virtual region of the video data and a second waveform monitor corresponding to a real region are displayed. When region information is detected from video data, a waveform monitor of the entire region of the video data (image data) may also be displayed. In this case, the display device further has one generation means for generating a waveform monitor. This generation means analyzes pixels for the entire region of the image data. The analysis result (third analysis data) is then generated as a third waveform monitor.

[0036] <Third embodiment> Hereinafter, an embodiment of the present invention will be described. An example will be described in which a display device has an image analysis device according to the present embodiment built in, but the image analysis device according to the present embodiment may be a device (such as a personal computer) separate from the display device.

[0037] Fig. 6 is a block diagram showing an example of the configuration of a display device according to this embodiment. The display device 600 in Fig. 6 includes a receiving unit 601, a receiving unit 602, a green screen composition unit 609, a control unit 603, a memory unit 104, a composition unit 105, a display unit 106, a generation unit 607, and a generation unit 608.

[0038] The receiving unit 601 acquires the first input moving image data and outputs the first input moving image data to the green screen composition unit 609. The first input moving image data is moving image captured of a performer performing in front of a green screen.

[0039] Receiving unit 602 acquires the second input moving image data and outputs the second input moving image data to green screen composition unit 609. The second input moving image data is moving image that has been shot separately from the first input moving image data.

[0040] In this embodiment, the receiving units 601 and 602 acquire frame data for each frame of a moving image from an external device. The receiving units 601 and 602 are input terminals that comply with standards such as SDI and HDMI (registered trademark). The external device may be an imaging device, a playback device, or the like.

[0041] The green screen composition unit 609 combines the first input video data acquired from the receiving unit 601 with the second input video data acquired from the receiving unit 602 to generate video data. Here, the video data is composed of pixel values ​​acquired from the second video data for pixels with a green color in the first video data, and pixel values ​​acquired from the first video data for other pixels. The green screen composition unit 609 also generates region information and outputs it to the control unit 603, the generation units 607, and the generation units 608. Here, the region information is information in which pixels with a green color in the first input video data are assigned as a virtual region, and other pixels are assigned as a real region. The green screen composition unit 609 can also individually adjust the image quality (brightness and color) of the first input video data and the second input video data.

[0042] The control unit 603 controls the processing of each unit of the display device 600. The control unit 603 is connected to each unit via a control bus, which is not shown in the figure to avoid complexity. For example, the control unit 603 is an arithmetic processing circuit that executes a program stored in the memory unit 104 to control the processing of each block of the display device 600. In this embodiment, the control unit 603 controls the composition unit 105 based on area information output from the green screen composition unit 609.

[0043] The memory unit 104 stores programs, parameters, etc. The programs and parameters stored in the memory unit 104 are read and written by each block of the display device 600.

[0044] The synthesis unit 105 synthesizes the moving image data with the first waveform monitor output by the generation unit 607 and the second waveform monitor output by the generation unit 608 based on the control content output from the control unit 603, to generate synthesized moving image data. The synthesis unit 105 then outputs the synthesized moving image data to the display unit 106.

[0045] Display unit 106 displays on a display surface a moving image based on the composite moving image data output from composition unit 105. Display unit 106 is, for example, a liquid crystal display unit having a liquid crystal panel and a backlight unit, or an organic EL display panel.

[0046] The generation unit 607 acquires the video data output from the green screen synthesis unit 609, and also acquires the area information output from the green screen synthesis unit 609. The generation unit 607 then generates a waveform monitor (first waveform monitor) related to pixels in the video data that are designated as a virtual area by the area information. More specifically, the generation unit 607 extracts one frame's worth of data from the video data, and further extracts pixels designated as a virtual area from that data, analyzes them, and generates the results as the first waveform monitor.

[0047] The generation unit 608 acquires the video data output from the green screen synthesis unit 609, and also acquires the area information output from the green screen synthesis unit 609. The generation unit 608 then generates a waveform monitor (second waveform monitor) related to pixels in the video data that are designated as real areas by the area information. More specifically, the generation unit 608 extracts one frame's worth of data from the video data, and further extracts pixels designated as real areas from that data, and generates the results of analyzing these as the second waveform monitor.

[0048] Having explained the operation of each unit, we will now explain the procedure for integrating these operations to display the first waveform monitor and the second waveform monitor on the display unit 106. Receiving unit 601 receives the first input video data, and receiving unit 602 receives the second input video data. Green screen composition unit 609 combines the first input video data and the second input video data to generate video data and area information, and outputs these to control unit 603 and generating units 607 and 608. Generators 607 and 608 refer to the area information to generate the first waveform monitor and the second waveform monitor. Combining unit 105 combines the first waveform monitor and the second waveform monitor with video data, and outputs this to display unit 106 as combined video data. Display unit 106 displays the combined video data including the first waveform monitor and the second waveform monitor.

[0049] 7A shows an example of the first input video data. A performer 703 is positioned in front of a green screen 701.

[0050] 7B is a diagram showing an example of second input video data, which is a video of an outside scene 711 and standing trees 712.

[0051] Fig. 8 shows an example of video data. A performer 703 is cut out from the first input video data shown in Fig. 7(A) and composited onto the second input video data shown in Fig. 7(B). Therefore, the video data is composed of the performer 703, an outside scene 711, and standing trees 712.

[0052] FIG. 9 is a diagram showing an example of the display of composite video data. Waveform monitor 901 is a first waveform monitor generated by generation unit 607, which visualizes the characteristics by analyzing the virtual region of the video data shown in FIG. 8. Waveform monitor 902 is a second waveform monitor generated by generation unit 608, which visualizes the characteristics by analyzing the real region of the video data shown in FIG. 8. Items that are the same as those in FIG. 8 are denoted by the same reference numerals as in FIG. 8. Waveform monitors 901 and 902 are synthesized with the video data shown in FIG. 8 to produce the composite video data shown in FIG. 9.

[0053] This embodiment eliminates the need for the user to specify the areas when analyzing the virtual area and the real area separately. By referring to the analysis results of the first and second waveform monitors, the user can easily check the brightness distribution of the virtual area and the real area. Furthermore, when adjusting the image quality for green screen compositing, the user can easily adjust the image quality difference between the first input video data and the second input video data (for example, to make the brightness of the first input video data and the second input video data approximately uniform).

[0054] [Other embodiments] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0055] The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer that can execute the program directly from a recording medium or via wired / wireless communication, and the program is executed.

[0056] Therefore, the program code itself that is supplied to and installed on a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0057] In this case, as long as it has the functionality of a program, the form of the program does not matter, such as object code, a program executed by an interpreter, or script data supplied to an OS.

[0058] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0059] Another method of supplying the program is to store the computer program forming the present invention in a server on a computer network, and have connected client computers download the computer program.

Claims

1. receiving means for receiving image data; a detection means for detecting a first image area corresponding to a virtual area and a second image area corresponding to a real area from the image data; a first generation means for analyzing the first image region and generating first analysis data; second generation means for analyzing the second image region and generating second analysis data; a display means for displaying the first analysis data and the second analysis data together with the image data; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the display means displays the first analysis data and the second analysis data superimposed on the image data.

3. Further, a third generation means is provided for analyzing the entire area of ​​the image data and generating third analysis data, 2. The image processing apparatus according to claim 1, wherein the display means displays the third analysis data together with the image data.

4. the display means displays the first analysis data, the second analysis data, and the third analysis data together with the image data when the detection means detects the first image area and the second image area; The display means displays the image data and the third analysis data when the detection means has not detected the first image area and the second image area.

4. The image processing device according to claim 3.

5. 5. The image processing device according to claim 1, wherein the detection means detects the first image area and the second image area based on identification information of the virtual area and the real area included in metadata related to the image data.

6. 6. The image processing apparatus according to claim 5, wherein the identification information is generated from distance information of the image data.

7. 7. The image processing apparatus according to claim 1, wherein the image data is image data generated by combining a plurality of images.

8. 8. The image processing apparatus according to claim 1, wherein the display means displays the first analysis data and the second analysis data as image signal levels relative to a horizontal resolution of the image.

9. 8. The image processing apparatus according to claim 1, wherein the display means displays the first analysis data and the second analysis data on a vectorscope.

10. a receiving step of receiving image data; a detecting step of detecting a first image area corresponding to a virtual area and a second image area corresponding to a real area from the image data; a first generation step of analyzing the first image region and generating first analysis data; a second generation step of analyzing the second image region and generating second analysis data; a display step of displaying the first analysis data and the second analysis data together with the image data; 1. A method for controlling an image processing apparatus, comprising:

11. a receiving step of receiving image data; a detecting step of detecting a first image area corresponding to a virtual area and a second image area corresponding to a real area from the image data; a first generation step of analyzing the first image region and generating first analysis data; a second generation step of analyzing the second image region and generating second analysis data; a display step of displaying the first analysis data and the second analysis data together with the image data; A program that causes a computer to execute the following.

Images