Signal processing device and signal processing device control method as well as program

Abstract

To provide a signal processing device which can easily analyze a virtual region and a real region separately and is suitable for virtual production, and a signal processing device control method.SOLUTION: The signal processing device comprises: first obtaining means that obtains image data that can be obtained when a photographing device photographs a subject and a background video displayed on a background display; first generating means that generates a first colored image in which a region including first frequency components of image data on a background video region is colored with a specific color showing a region including predetermined frequency components; second generating means that generates a second colored image in which a region including second frequency components of image data on a subject region is colored with a specific color showing the region including the predetermined frequency components; and synthesizing means that synthesizes the first colored image with the second colored image to output synthesized image data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a signal processing device, a control method for the signal processing device, and a program. [Background technology]

[0002] Virtual production of in-camera VFX using LED walls is becoming more common. An LED wall is a display device with LEDs arranged in a grid pattern. In in-camera VFX, a previously captured image is displayed on the LED wall, and performers and art sets are placed between the LED wall and the camera, and the LED wall, performers, and art sets are photographed. Of the images captured by in-camera VFX, the partial area in which the performers and art sets are photographed is referred to here as the real area (subject area), and the partial area in which the LED wall is photographed is referred to here as the virtual area (background image area).

[0003] The virtual area is an image taken by the first camera that is displayed on an LED wall and then photographed again by the second camera, whereas the real area is an image taken directly by the second camera.

[0004] If the focus is accidentally set on the LED wall, moire is likely to occur. On the other hand, it is necessary to focus precisely on subjects such as performers and art sets. Therefore, in virtual production, there is a demand to be able to observe the focus state of the virtual area and the real area separately.

[0005] For example, Patent Document 1 discloses a technique for limiting a subject having distance information substantially identical to that of a specific subject in an image and performing peaking processing. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-219920 A Summary of the Invention [Problem to be solved by the invention]

[0007] According to the method of Patent Document 1, it is possible to perform peaking processing on partial regions, but since it does not take into account virtual and real regions, it is not possible to intuitively distinguish between the virtual and real regions and grasp the characteristics of each region.

[0008] Therefore, the present invention provides a signal processing device and a signal processing method that can easily analyze a virtual area and a real area separately and are suitable for virtual production. [Means for solving the problem]

[0009] The technical features of the present invention include a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; an extraction step of extracting information on the subject region and the background image region in the image data; a first generation step of generating a first colored image in which a region including a first frequency component in the image data of the background image region is colored a specific color indicating that it is a region including the first frequency component; a second generation step of generating a second colored image in which a region including a second frequency component in the image data of the subject region is colored a specific color indicating that it is a region including the second frequency component; and a synthesis step of synthesizing the first colored image and the second colored image to output synthetic image data. Another technical feature of the present invention is characterized by having a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; an extraction step of extracting information on the subject region and the background image region in the image data; a first generation step of generating a first colored image in which an area of ​​a predetermined luminance level in a display image of the image data of the background image region is colored a specific color indicative of the area being an area of ​​the predetermined luminance level; a second generation step of generating a second colored image in which an area of ​​a predetermined luminance level in a display image of the image data of the subject region is colored a specific color indicative of the area being an area of ​​the predetermined luminance level; and a synthesis step of synthesizing the first colored image and the second colored image to output synthetic image data. Another technical feature of the present invention is characterized in that it comprises a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; an extraction step of extracting information on the subject area and the background image area in the image data; a first generation step of changing the color or brightness of a pixel of the image data of the background image area when the pixel has a chromaticity outside a predetermined color gamut; a second generation step of changing the color or brightness of the pixel of the image data of the subject area when the pixel has a chromaticity outside a predetermined color gamut; and a synthesis step of synthesizing the image data output by the first generation step and the image data output by the second generation step to output synthetic image data. Effect of the Invention

[0010] According to the present invention, it is possible to provide a signal processing device and a signal processing method that are suitable for virtual production, by simply and easily analyzing a virtual area and a real area separately. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of a signal processing device according to first and second embodiments. [Diagram 2] 1 is a block diagram showing an example of the configuration of an imaging device according to first, second and third embodiments. [Diagram 3] FIG. 2 is a block diagram showing an example of device connections according to the first, second and third embodiments. [Figure 4] FIG. 1 is a diagram showing an outline of in-camera VFX shooting from a bird's-eye view according to the first, second, and third embodiments. [Diagram 5] FIG. 2 is a diagram showing an example of captured video data according to the first, second and third embodiments. [Figure 6] FIG. 2 is a diagram showing an example of display of composite moving image data according to the first embodiment. [Figure 7] FIG. 11 is a diagram showing a luminance distribution in an example of captured moving image data according to the second embodiment. [Figure 8] 10 is a table showing colors in a false color display corresponding to the luminance of captured video data according to the second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of display of composite moving image data according to the second embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of a signal processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] (First embodiment) ●Signal processing device configuration Fig. 1 is a block diagram showing an example of the configuration of a signal processing device according to this embodiment. The signal processing device 100 in Fig. 1 includes a receiving unit 101, a detecting unit 102, a control unit 103, a memory unit 104, a synthesizing unit 105, a display unit 106, a generating unit 107, and a generating unit 108.

[0014] The receiving unit 101 acquires captured video data 109, and outputs the video data to the detecting unit 102, the synthesizing unit 105, the generating unit 107, and the generating unit 108. In this embodiment, the receiving unit 101 acquires image data for each frame of a video from an external device. The receiving unit 101 then outputs the acquired image data to each unit in the subsequent stage. The receiving unit 101 is an input terminal that complies with standards such as SDI and HDMI (registered trademark). The external device is an imaging device, a playback device, or the like. In this embodiment, an example of connection to an imaging device 200 will be described, and details will be described later.

[0015] The detection unit 102 acquires the video data including the region information output from the reception unit 101, and outputs the acquired region information to the control unit 103, the generation unit 107, and the generation unit 108. The region information will be described later.

[0016] The control unit 103 controls the processing of each unit of the signal processing device 100. The control unit 103 is connected to each unit via a control bus, but the control bus is not shown in the figure because it is complicated. For example, the control unit 103 is an arithmetic processing circuit that executes a program stored in the memory unit 104 to control the processing of each block of the signal processing device 100. In this embodiment, the control unit 103 controls the synthesis unit 105 based on the region information output from the detection unit 102.

[0017] The memory unit 104 stores programs, parameters, etc. The programs and parameters stored in the memory unit 104 are read and written by each block of the signal processing device 100.

[0018] The synthesis unit 105 synthesizes the first peaking display output by the generation unit 107 and the second peaking display output by the generation unit 108 with the video data based on the control content output from the control unit 103, to generate synthesized video data. Then, the synthesis unit 105 outputs the synthesized video data to the display unit 106.

[0019] The display unit 106 displays on a display surface a moving image based on the composite moving image data output from the composition unit 105. The display unit 106 is, for example, a display panel configured with a liquid crystal panel and a backlight unit, an organic EL display panel, or the like.

[0020] The generating unit 107 acquires the video data output from the receiving unit 101 and the region information output from the detecting unit 102, and generates a peaking display (first peaking display) for pixels designated as a virtual region by the region information in the video data. More specifically, one frame's worth of data is extracted from the video data, and subject edges are detected from the signals of pixels designated as a virtual region based on an edge detection frequency band, and the analysis result is generated as a first peaking display. This makes it possible to create a display image including a colored image in which a region containing a predetermined frequency component in the display image is colored in a specific color indicating that the region contains the predetermined frequency component.

[0021] Generation unit 108 acquires the video data output from receiving unit 101 and also acquires the area information output from detection unit 102, and generates a peaking display (second peaking display) for pixels designated as a real area by the area information in the video data. More specifically, one frame's worth of data is extracted from the video data, and subject edges are detected from the data for pixels designated as a real area based on the edge detection frequency band, and the analysis result is generated as the second peaking display.

[0022] The operation of each unit has been described, but the procedure for displaying the first peaking display and the second peaking display on the display unit 106 by integrating the operations will now be described. When the receiving unit 101 receives captured video data 109, the detecting unit 102 extracts region information from the video data and outputs it to the control unit 103, the generating unit 107, and the generating unit 108. The generating units 107 and 108 refer to the region information and generate the first peaking display and the second peaking display. The combining unit 105 combines the first peaking display and the second peaking display with the video data and outputs it to the display unit 106 as combined video data. The display unit 106 displays the combined video data including the first peaking display and the second peaking display.

[0023] Imaging device configuration 2 is a block diagram showing an example of the configuration of an image capturing device that generates captured video data acquired by a signal processing device according to this embodiment. The image capturing device 200 in FIG. 2 includes an optical unit 201, an image capturing unit 202, a region information superimposing unit 203, a transmitting unit 204, and a distance measuring unit 205.

[0024] The optical unit 201 includes a lens, a focus motor, and the like, and forms an image of an object to be photographed on the imaging unit 202. The imaging unit 202 includes an image sensor, and the like, and generates video data from the image formed by the optical unit 201, and outputs the video data to the area information superimposing unit 203. The area information superimposing unit 203 superimposes area information on the video data, and outputs the video data to the transmitting unit 204. The area information will be described later. The transmitting unit 204 transmits the video data received from the area information superimposing unit 203 as the captured video data 109. The transmitting unit 204 is an output terminal that complies with standards such as SDI and HDMI. The distance measuring unit 205 measures the distance between the imaging device and the object to be photographed for each pixel on the image sensor. The distance measured by the distance measuring unit 205 is used to generate area information, and the details will be described later. There is known a technology for measuring distance by arranging pixels that detect phase differences on an image sensor, and in this case, the imaging unit 202 and the distance measuring unit 205 are integrated into one unit.

[0025] ● Device connection example 3 is a block diagram showing an example of connections between an image generating device, a display device, an imaging device, and a signal processing device. The image generating device 400 generates moving image data by storing pre-captured moving image data inside and playing it back. The image generating device 400 outputs the generated moving image data to the display device 300 as generated moving image data 301. The display device 300 is an LED wall, and is a background display that displays the generated moving image data 301 (background video data). The imaging device 200 outputs the captured moving image data to the signal processing device 100 as captured moving image data 109.

[0026] Overview of the shoot FIG. 4 is a diagram showing an overview of in-camera VFX shooting from a bird's-eye view. Actor 503 performs between display device 300 and imaging device 200. Imaging device 200 captures display device 300 and actor 503. The dotted line indicates the angle of view of optical unit 201. Note that while FIG. 4 shows a case where there is one actor who is the subject to be captured, there may be multiple actors, or an art set in addition to the actors. A background image is displayed on display device 300 and captured by imaging device 200.

[0027] - Area information to distinguish between real and virtual areas Here, the area information will be described. The area information is information for identifying whether each pixel is a real area or a virtual area. The area information superimposing unit 203 assigns area information to pixels whose distance measured by the distance measuring unit 205 is approximately equal to the distance between the imaging device 200 and the display device 300 as a virtual area, and other pixels as a real area. The distance between the imaging device 200 and the display device 300 is measured by the users of the imaging device 200 and the display device 300 and set in the imaging device 200. The area information is superimposed on the video data as metadata. Examples of the metadata include ANC data standardized in SDI and InfoFrame standardized in HDMI. In the ANC data, any binary data can be superimposed on the horizontal blanking period. Although a specific method for superimposing the area information is not standardized, the area information can be superimposed by using the ANC data as follows. A 1-bit value 1 indicates that one pixel is a virtual area, and a 1-bit value 0 indicates that one pixel is a real area. Data summarizing the values ​​(1 or 0) of the number of pixels for one line is generated and superimposed as ANC data during the horizontal blanking period of the corresponding line.

[0028] ●An example of video data FIG. 5 is a diagram showing an example of shot video data 109. Wall 501 and window frame 502 are an art set. A performer 503 is positioned in front of wall 501 of the art set. Outside view 511 and standing tree 512 are positioned inside window frame 502. Outside view 511 and standing tree 512 are photographed in advance and displayed on display device 300. In this state, when wall 501, window frame 502, performer 503, and display device 300 are photographed, the shot video data shown in FIG. 5 is obtained. Wall 501, window frame 502, and performer 503 are real areas. Outside view 511 and standing tree 512 are virtual areas.

[0029] ●An example of composite video data displayed on the display unit (peaking display) FIG. 6 is a diagram showing an example of the composite video data displayed by the display unit 106. The generation unit 107 generates a first peaking display (peaking display in the virtual area) in a first display form. The generation unit 108 generates a second peaking display (peaking display in the real area) in a second display form different from the first display form. In the example shown in FIG. 6, the first form is light gray and the second form is dark gray. The tree 512 is a shooting object in the virtual area, so the generation unit 107 applies a peaking display in the first display form (light gray) (reference number 601). The performer 503 is a shooting object in the real area, so the generation unit 108 applies a peaking display in the second display form (dark gray) (reference number 602). The window frame 502 is a shooting object in the real area, but is located at the boundary with the virtual area. Therefore, the virtual area side of the boundary is peaked in a first display form (light gray) by the generation unit 107, and the real area side of the boundary is peaked in a second display form (dark gray) by the generation unit 108 (symbol 603).

[0030] The first peaking display and the second peaking display are combined with the captured moving image data shown in Fig. 5 to produce the combined moving image data shown in Fig. 6. The combined moving image data shown in Fig. 6 is displayed by the display unit 106.

[0031] Effects of this embodiment This embodiment makes it easy to apply peaking display separately to the virtual area and the real area, and provides a guide display that allows the user to intuitively distinguish between the virtual area and the real area and grasp the focus state of each area.

[0032] Variations In the present embodiment, the signal processing device 100 receives moving image data, but may receive still image data. In the case of moving images, the receiving unit 101 acquires frame data for each frame of the moving image, while in the case of still images, the receiving unit 101 acquires new still image data every time the still image data is changed. Then, composite image data is generated and displayed.

[0033] In the present embodiment, the display unit 106 displays the composite moving image data. The composite moving image data may be output to the outside using SDI or HDMI.

[0034] Moreover, the generating unit 107 and the generating unit 108 generate a peaking display by displaying high-frequency components of the spatial frequency of the video data in the first display form or the second display form. In this case, it is preferable that the range of spatial frequencies extracted by the generating unit 107 as high-frequency components is extracted to a lower frequency side than that of the generating unit 108. For example, the generating unit 107 extracts components of spatial frequencies equal to or higher than 1 / 16 of the sampling frequency and sets them as the target of the peaking display, and the generating unit 108 extracts components of spatial frequencies equal to or higher than 1 / 4 of the sampling frequency and sets them as the target of the peaking display. As a result, the first peaking display, which is a peaking display in the virtual region, is a peaking display target for a lower frequency side than the second peaking display, which is a peaking display in the real region. Since the video data in the virtual region is generated by photographing the LED wall, moire due to the pixels of the LED wall is a concern. By setting the target of the peaking display to a lower frequency side in the virtual region, moire due to focusing on the LED wall against the user's intention can be reduced.

[0035] Furthermore, the generating units 107 and 108 may each be provided with an operation unit for allowing a user to set a spatial frequency to be the target of peaking display, independently of each other. Since the spatial frequency at which moire generation is a concern varies depending on the number of pixels of the imaging device and the LED wall, by allowing the user to set an appropriate spatial frequency, it is possible to reduce moire caused by focusing on the LED wall against the user's intention.

[0036] Furthermore, when the generation unit 107 detects a frequency component that is a target of peaking display and the frequency component is equal to or greater than a predetermined value, the signal processing device 100 may issue an instruction to the imaging device 200 to control the processing to open the aperture of the optical unit 201. By opening the aperture, it is possible to reduce moire caused by focusing on the LED wall. Although not shown in FIG. 3, it is assumed that the signal processing device 100 and the imaging device 200 are capable of communicating with each other via a LAN (Local Area Network). Furthermore, the aperture is opened when the frequency component that is a target of peaking display extracted by the generation unit 107 is equal to or greater than a certain level. For example, it is assumed that the number of pixels that are the target of peaking display is equal to or greater than 10% of the number of pixels in the virtual region.

[0037] Also, when the generating unit 107 extracts the frequency components to be the target of the peaking display, the signal processing device 100 may notify the image generating device 400 of an instruction to perform a blurring process on the generated video data 301. The blurring process refers to, for example, performing a low-pass filter process to reduce high-frequency components. By performing the blurring process, it is possible to reduce moire caused by focusing on the LED wall. Although not shown in FIG. 3, it is assumed that the signal processing device 100 and the image generating device 400 can communicate with each other via a LAN. Also, the high-frequency components are reduced when the frequency components to be the target of the peaking display extracted by the generating unit 107 are equal to or greater than a certain level. For example, it is assumed that the number of pixels to be the target of the peaking display is equal to or greater than 10% of the number of pixels in the virtual area.

[0038] In addition, in order to easily check the focus state of the LED wall, an image for checking the focus state may be displayed on the LED wall. In this case, it is preferable to apply the present invention as described below. The signal processing device 100 is configured to be able to switch between enabling and disabling the peaking display. The signal processing device 100 sends an instruction to the image generating device 400, and the image generating device 400 outputs an image for checking the focus as generated video data 301. The image for checking the focus is, for example, an image in which white horizontal lines and white vertical lines are added to a black background. Furthermore, the signal processing device 100 switches the peaking display to be enabled. The display device 300 displays the image for checking the focus. The imaging device 200 captures the display device 300 displaying the image for checking the focus. Furthermore, the signal processing device 100 applies the peaking display to the captured video data 109 generated by capturing the display device 300 displaying the image for checking the focus. By checking the first peaking display applied to the image for checking the focus, the focus state of the LED wall can be known. Although not shown in FIG. 3, it is assumed that the signal processing device 100 and the image generating device 400 are capable of communicating with each other via a LAN.

[0039] Second Embodiment The second embodiment of the present invention will be described below. However, since this embodiment is a modified version of the first embodiment, the description will focus on the differences from the first embodiment, and the same content as the first embodiment will be omitted as appropriate.

[0040] ●Signal processing device configuration The receiving unit 101, the detecting unit 102, the control unit 103, the memory unit 104, and the display unit 106 are the same as those in the first embodiment.

[0041] In the first embodiment, the synthesis unit 105 synthesizes the first peaking display and the second peaking display with the video data to generate synthesized video data. Meanwhile, in the second embodiment, the synthesis unit 105 synthesizes the first false color display and the second false color display with the video data to generate synthesized video data. Also, in the first embodiment, the generation units 107 and 108 generate the first peaking display and the second peaking display, but in the second embodiment, the generation units 107 and 108 generate the first false color display and the second false color display. This will be explained in detail below.

[0042] The synthesis unit 105 synthesizes the video data with the first false color display output by the generation unit 107 and the second false color display output by the generation unit 108 based on the control content output by the control unit 103, to generate synthesized video data. The synthesis unit 105 then outputs the synthesized video data to the display unit 106.

[0043] The generating unit 107 acquires the video data output from the receiving unit 101, and also acquires the area information output from the detecting unit 102. Then, it generates a false color display (first false color display) for pixels in the video data that are specified as virtual areas by the area information. More specifically, it extracts one frame's worth of data from the video data, and further processes the gradation value of each pixel from that data so that it is displayed in a predetermined color according to the luminance of the pixel specified as the virtual area, and generates the result as the first false color display. In this way, it is possible to create a display image that includes a colored image in which an area of ​​a predetermined luminance level in the display image is colored in a specific color that indicates that it is an area of ​​the predetermined luminance level.

[0044] Generation unit 108 acquires the video data output from receiving unit 101, and also acquires the region information output from detection unit 102. Then, generation unit 108 generates a false color display (second false color display) for pixels in the video data that are specified as real regions by the region information. More specifically, one frame's worth of data is extracted from the video data, and processing is performed to convert the gradation value of each pixel so that it is displayed in a predetermined color according to the luminance of the pixels specified as real regions from that data, and the result is generated as the second false color display.

[0045] The operation of each unit has been described, but the procedure for displaying the first and second false color displays on display unit 106 by integrating those operations will now be described. When receiving unit 101 receives captured video data 109, detection unit 102 extracts region information from the video data and outputs it to control unit 103 and generation units 107 and 108. Generation units 107 and 108 refer to the region information and generate the first and second false color displays. Synthesis unit 105 synthesizes the first and second false color displays with the video data, and outputs the synthesized video data to display unit 106. Display unit 106 displays the synthesized video data including the first and second false color displays.

[0046] ●Brightness distribution in an example of captured video data Fig. 7 is a diagram showing the brightness distribution in an example of the captured video data 109 shown in Fig. 5. The upper part of the external scene 511 has a brightness of 1000 nit (reference number 701), the middle part has a brightness of 200 nit (reference number 702), and the lower part has a brightness of 20 nit (reference number 703). The upper part of the standing tree 512 has a brightness of 100 nit (reference number 704), and the lower part has a brightness of 60 nit (reference number 705). The upper part of the wall 501 has a brightness of 500 nit (reference number 706), the middle part has a brightness of 200 nit (reference number 707), and the lower part has a brightness of 20 nit (reference number 708). The upper part of the performer 503 has a brightness of 180 nit (reference number 709), and the lower part has a brightness of 100 nit (reference number 710). The window frame 502 has a brightness of 60 nit (reference number 711).

[0047] ● A table showing the colors of the false color display corresponding to the brightness of the captured video data Fig. 8(A) is a table showing the colors of the false color display associated with the luminance of the captured video data when the generating unit 107 generates the first false color display. Fig. 8(B) is a table showing the colors of the false color display associated with the luminance of the captured video data when the generating unit 108 generates the second false color display. The luminance range of the captured video data is from 0 nit to 1000 nit.

[0048] An example of composite video data displayed on the display unit (false color display) Fig. 9 is a diagram showing an example of the composite moving image data displayed by the display unit 106. The first false color display and the second false color display are composited with the captured moving image data shown in Fig. 5 to produce the composite moving image data shown in Fig. 9. The composite moving image data shown in Fig. 9 is displayed by the display unit 106.

[0049] Since the external scene 511 and the standing tree 512 are the shooting subjects in the virtual area, they are colored by the first false color display generated by the generation unit 107 according to Fig. 8(A). The upper part of the external scene 511 has a luminance of 1000 nits and is colored light red (reference number 901), the middle part has a luminance of 200 nits and is colored light green (reference number 902), and the lower part has a luminance of 20 nits and is colored light blue (reference number 903). The upper part of the standing tree 512 has a luminance of 100 nits and is colored light cyan (reference number 904), and the lower part has a luminance of 60 nits and is colored light cyan (reference number 905).

[0050] Since the wall 501, window frame 502, and performer 503 are objects to be photographed in the real area, they are colored by the second false color display generated by the generation unit 108 according to Fig. 8(B). The upper part of the wall 501 has a luminance of 500 nits and is therefore colored yellow (reference number 906), the middle part has a luminance of 200 nits and is therefore colored green (reference number 907), and the lower part has a luminance of 20 nits and is therefore colored blue (reference number 908). The upper part of the performer 503 has a luminance of 180 nits and is therefore colored green (reference number 909), and the lower part has a luminance of 100 nits and is therefore colored cyan (reference number 910). The window frame 502 has a luminance of 60 nits and is therefore colored cyan (reference number 911).

[0051] By coloring the virtual area according to FIG. 8(A) and the real area according to FIG. 8(B), if the brightness is the same in the virtual area and the real area, they are colored with colors of the same hue but different intensities.

[0052] Effects of this embodiment According to this embodiment, it is easy to display the virtual area and the real area separately in false color, and by changing the density of the color used for coloring, it is possible to intuitively distinguish the virtual area from the real area and grasp the luminance of each area. Moreover, by using the same hue for coloring for the same luminance, it is possible to intuitively grasp the parts of the virtual area and the real area that are similar in luminance.

[0053] Variations In this embodiment, an example has been described in which the brightness range of the video data from 0 nit or more to 1000 nit or less is colored in one of the colors according to Fig. 8. However, it is also possible not to color a part of the brightness range. For example, in the case of the brightness range from 0 nit or more to 50 nit, it is also possible to display the video data itself on the display unit 106 without coloring it.

[0054] (Third embodiment) The third embodiment of the present invention will be described below. However, since this embodiment is a modified version of the first and second embodiments, the description will focus on the differences from the first embodiment, and the content similar to the first and second embodiments will be omitted as appropriate.

[0055] ●Signal processing device configuration FIG. 10 is a block diagram showing an example of the configuration of a signal processing device according to this embodiment.

[0056] The synthesis unit 105 synthesizes the first peaking display or the first false color display, and the second peaking display or the second false color display with the video data based on the control contents output from the control unit 105 to generate synthesized video data. Then, the synthesis unit 105 outputs the synthesized video data to the display unit 106.

[0057] The generating unit 107 acquires the video data output from the receiving unit 101, and also acquires the region information output from the detecting unit 102. Then, the generating unit 107 generates a peaking display (first peaking display) or a false color display (first false color display) for pixels designated as a virtual region by the region information in the video data. More specifically, one frame's worth of data is extracted from the video data, pixels designated as a virtual region are extracted from that data, and the results of analyzing these are generated as the first peaking display or the first false color display. Furthermore, the generating unit 107 selects and generates the first peaking display or the first false color display in accordance with the method selecting unit 1001.

[0058] Generation unit 108 acquires the video data output from receiving unit 101, and also acquires the region information output from detecting unit 102. Then, it generates a peaking display (second peaking display) or a false color display (second false color display) for pixels designated as real regions by the region information in the video data. More specifically, it extracts one frame's worth of data from the video data, and further extracts pixels designated as real regions from that data, and generates the results of analyzing them as the second peaking display or the second false color display. Furthermore, generation unit 108 selects and generates the second peaking display or the second false color display in accordance with method selection unit 1001.

[0059] The method selection unit 1001 selects peaking display or false color display in accordance with a user's operation, and notifies the generation unit 107 and the generation unit 108 of the selection.

[0060] When the method selection unit 1001 selects the peaking display, the generation unit 107 generates a first peaking display, and the generation unit 108 generates a second peaking display. Furthermore, the synthesis unit 105 synthesizes the first peaking display and the second peaking display, and the display unit 106 displays the synthesized result.

[0061] When the method selection unit 1001 selects the false color display, the generation unit 107 generates a first false color display, and the generation unit 108 generates a second false color display. Furthermore, the composition unit 105 composes the first false color display and the second false color display, and the display unit 106 displays the composite image.

[0062] Effects of this embodiment According to this embodiment, the user can select peaking display or false color display by operation, so that the features desired by the user can be visualized. This allows the features of the virtual area and the real area to be grasped. This makes it easy to confirm the difference in image quality between the virtual area and the real area and to adjust the image quality.

[0063] (Fourth embodiment) Hereinafter, a fourth embodiment of the present invention will be described. However, since this embodiment is a modified version of the first to third embodiments, the description will focus on the differences from the first to third embodiments, and the content similar to the first to third embodiments will be omitted as appropriate.

[0064] ●Out of color gamut warning display The above describes an example in which the present invention is applied to peaking display and false color display. The present invention can also be applied to other image analysis methods. For example, the present invention can also be applied to a display for out-of-gamut warning. The display for out-of-gamut warning is an image analysis method in which, when a pixel of captured video data has a chromaticity outside a specified color gamut (for example, a color gamut defined in ITU-R BT.709), the color or brightness of the pixel is changed and displayed. The out-of-gamut pixel may be displayed in a different manner, for example, red in the virtual area and blue in the real area.

[0065] The above embodiment includes the following configurations.

[0066] (Configuration 1) a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for generating a first colored image by coloring a region including a first frequency component of the image data of the background image region with a specific color indicating that the region includes the first frequency component; a second generating means for generating a second colored image by coloring a region including a second frequency component of the image data of the subject region with a specific color indicating that the region includes the second frequency component; a synthesizing unit that synthesizes the first colored image and the second colored image and outputs synthesized image data.

[0067] (Configuration 2) 2. The signal processing device according to configuration 1, wherein the first frequency component is a lower frequency than the second frequency component.

[0068] (Configuration 3) 3. The signal processing device according to configuration 1 or 2, wherein the specific color differs between the first generating means and the second generating means.

[0069] (Configuration 4) 4. The signal processing device according to any one of configurations 1 to 3, wherein the first generating means and the second generating means generate a peaking display.

[0070] (Configuration 5) The signal processing device according to configuration 4, further comprising a notification means for notifying the imaging device of an instruction to open an aperture when the number of pixels to be the target of peaking display generated by the first generation means is equal to or greater than a predetermined value.

[0071] (Configuration 6) The signal processing device according to configuration 4, further comprising a notification means for notifying an image generating device that generates background video data of an instruction to perform blurring processing when the number of pixels that are the subject of peaking display generated by the first generating means is equal to or greater than a predetermined value.

[0072] (Configuration 7) 5. The signal processing device according to configuration 4, further comprising a notification means for notifying an image generating device that generates background image data of an instruction to display an image for focus confirmation.

[0073] (Configuration 8) a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for generating a first colored image by coloring an area of ​​a predetermined luminance level in a display image of the image data of the background image area with a specific color indicating that the area is an area of ​​the predetermined luminance level; a second generating means for generating a second colored image by coloring an area having a predetermined luminance level in a display image of the image data of the subject area with a specific color that indicates that the area has the predetermined luminance level; a synthesizing unit that synthesizes the first colored image and the second colored image and outputs synthesized image data.

[0074] (Configuration 9) 9. The signal processing device according to configuration 8, wherein for pixels having the same luminance of image data, the specific color has the same hue but different density between the first generation means and the second generation means.

[0075] (Configuration 10) 10. The signal processing device according to configuration 8 or 9, wherein the first generating means and the second generating means generate a false color display.

[0076] (Configuration 11) a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for changing a color or brightness of a pixel of the image data of the background image region when the pixel has a chromaticity outside a predetermined color gamut; a second generating means for changing a color or brightness of a pixel of the image data of the object region when the pixel has a chromaticity outside a predetermined color gamut; a synthesizing unit that synthesizes the image data output by the first generating unit and the image data output by the second generating unit and outputs synthesized image data.

[0077] (Configuration 12) 12. The signal processing device according to claim 11, wherein the first generating means and the second generating means generate a display for an out-of-gamut warning.

[0078] (Configuration 13) 13. The signal processing device according to any one of configurations 1 to 12, further comprising control means for controlling a display means to display the composite image data output by the synthesis means.

[0079] (Method 1) a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of generating a first colored image by coloring a region including a first frequency component of the image data of the background image region with a specific color indicating that the region includes the first frequency component; a second generation step of generating a second colored image by coloring a region including a second frequency component of the image data of the subject region with a specific color indicating that the region includes the second frequency component; a combining step of combining the first colored image and the second colored image and outputting combined image data.

[0080] (Method 2) a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of generating a first colored image by coloring an area with a predetermined luminance level in a display image of the image data of the background video area with a specific color indicating that the area has the predetermined luminance level; a second generation step of generating a second colored image by coloring an area having a predetermined luminance level in a display image of the image data of the subject area with a specific color that indicates that the area has the predetermined luminance level; a combining step of combining the first colored image and the second colored image and outputting combined image data.

[0081] (Method 3) a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of changing a color or brightness of a pixel of the image data of the background image region when the pixel has a chromaticity outside a predetermined color gamut; a second generation step of changing the color or brightness of a pixel of the image data of the object region when the pixel has a chromaticity outside a predetermined color gamut; a synthesis step of synthesizing the image data output by the first generation step and the image data output by the second generation step and outputting synthesized image data.

[0082] (Program 1) A program for causing a computer to execute each step of the control method according to any one of Methods 1 to 3.

[0083] (Other embodiments) Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0084] The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program directly from a recording medium or via wired / wireless communication, and the program is executed.

[0085] Therefore, the program code itself that is supplied to and installed in a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0086] In this case, so long as the program has the functions of a program, the form of the program is not important, such as object code, a program executed by an interpreter, or script data supplied to an OS.

[0087] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0088] As a method for supplying the program, a method is also conceivable in which the computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program. [Explanation of symbols]

[0089] 101 Receiving unit 102 Detection unit 103 Control section 104 Memory section 105 Synthesis section 106 Display section 107, 108 Generation part 1001 Method selection section

Claims

1. a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for generating a first colored image by coloring a region including a first frequency component of image data of the background image region with a specific color indicating that the region includes the first frequency component; a second generating means for generating a second colored image by coloring a region including a second frequency component of the image data of the subject region with a specific color indicating that the region includes the second frequency component; a synthesizing unit for synthesizing the first colored image and the second colored image and outputting synthesized image data.

2. 2. The signal processing apparatus according to claim 1, wherein the first frequency component is lower in frequency than the second frequency component.

3. 2. The signal processing device according to claim 1, wherein the specific color is different between the first generating means and the second generating means.

4. 2. The signal processing apparatus of claim 1, wherein the first generating means and the second generating means generate a peaking indication.

5. 5. The signal processing device according to claim 4, further comprising a notification means for notifying the imaging device of an instruction to open an aperture when the number of pixels to be the target of peaking display generated by the first generation means is equal to or greater than a predetermined value.

6. 5. The signal processing device according to claim 4, further comprising a notification means for notifying an image generating device that generates background video data of an instruction to perform blurring processing when the number of pixels that are the subject of peaking display generated by the first generating means is equal to or greater than a predetermined value.

7. 5. The signal processing device according to claim 4, further comprising a notification unit for notifying an image generating device that generates background image data of an instruction to display an image for focus confirmation.

8. a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for generating a first colored image by coloring an area having a predetermined luminance level in a display image of the image data of the background image area with a specific color that indicates that the area has the predetermined luminance level; a second generating means for generating a second colored image by coloring an area having a predetermined luminance level in a display image of the image data of the subject area with a specific color that indicates that the area has the predetermined luminance level; a synthesizing unit for synthesizing the first colored image and the second colored image and outputting synthesized image data.

9. 9. The signal processing device according to claim 8, wherein for pixels having the same luminance of image data, the specific color has the same hue but different density in the first generation means and the second generation means.

10. 9. The signal processing apparatus of claim 8, wherein the first generating means and the second generating means generate a false color representation.

11. a first acquisition means for acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction means for extracting information on a background image region and a subject region from the image data; a first generating means for changing a color or brightness of a pixel of the image data of the background image region when the pixel has a chromaticity outside a predetermined color gamut; a second generating means for changing a color or brightness of a pixel of the image data of the object region when the pixel has a chromaticity outside a predetermined color gamut; a synthesizing means for synthesizing the image data output by said first generating means and the image data output by said second generating means, and outputting synthesized image data.

12. The signal processing apparatus of claim 11 , wherein the first generating means and the second generating means generate an indication for an out-of-gamut warning.

13. 13. The signal processing apparatus according to claim 1, further comprising a control unit for controlling a display unit to display the composite image data output by the combining unit.

14. a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of generating a first colored image by coloring a region including a first frequency component of image data of the background image region with a specific color indicating that the region includes the first frequency component; a second generation step of generating a second colored image by coloring a region including a second frequency component of the image data of the subject region with a specific color indicating that the region includes the second frequency component; a combining step of combining the first color image and the second color image and outputting combined image data.

15. a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of generating a first colored image by coloring an area having a predetermined luminance level in a display image of the image data of the background video area with a specific color that indicates that the area has the predetermined luminance level; a second generation step of generating a second colored image by coloring an area having a predetermined luminance level in a display image of the image data of the subject area with a specific color that indicates that the area has the predetermined luminance level; a combining step of combining the first color image and the second color image and outputting combined image data.

16. a first acquisition step of acquiring image data obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; an extraction step of extracting information on a subject area and a background image area in the image data; a first generation step of changing a color or brightness of a pixel of the image data of the background image region when the pixel has a chromaticity outside a predetermined color gamut; a second generation step of changing the color or brightness of a pixel of the image data of the object region when the pixel has a chromaticity outside a predetermined color gamut; a synthesis step of synthesizing the image data output by the first generation step and the image data output by the second generation step and outputting synthesized image data.

17. A program for causing a computer to execute each step of the control method according to any one of claims 14 to 16.

Images