Signal processing device, control method, and program

Abstract

To notify a user of image quality difference of a boundary of the same object at boundaries between a real area and a virtual area.SOLUTION: A signal processing device comprises: receiving means for receiving image data; detection means for detecting a first area corresponding to a virtual area and a second area corresponding to a real area from the image data; and display means for displaying a boundary as the boundary at which both of the first area and the second area show the same object of boundaries between the first area and the second area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a signal processing device used in in-camera VFX. [Background technology]

[0002] Virtual production of in-camera VFX using LED walls is becoming more common.

[0003] In virtual production, the same objects and stages as those displayed on the LED wall are prepared to create a visual connection between the real and virtual spaces. In such cases, differences in brightness and chromaticity can be seen at the boundary between the real and virtual areas of the filmed footage.

[0004] Patent Document 1 describes that in multi-projection using a plurality of projectors, the boundary between adjacent projected images is detected, and the difference in image quality at the boundary is measured and notified. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-159372 Summary of the Invention [Problem to be solved by the invention]

[0006] However, Patent Document 1 does not take into consideration the boundary between the projected image and the projection surface. As described above, in the past, the boundary between the area displayed on the display device and the area in real space was not taken into consideration, so when a user viewed a video created by virtual production, for example, the user might feel uncomfortable about the boundary. [Means for solving the problem]

[0007] The signal processing device includes a receiving means for receiving image data, a detecting means for detecting from the image data a first region corresponding to a virtual region and a second region corresponding to a real region, and a display means for displaying a boundary between the first region and the second region where both the first region and the second region represent the same object. [Effects of the Invention]

[0008] According to the present invention, the user can know the difference in image quality between the boundary of the same object at the boundary between the real area and the virtual area. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a display device according to a first embodiment. [Figure 2] 10 is a flowchart showing an example of an image quality difference notification process targeted at the boundary between a real area and a virtual area according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of frame image data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a detected boundary according to the first embodiment. [Figure 5] FIG. 4 is a diagram showing an area that is the target of averaging pixel values ​​at each coordinate according to the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a method for notifying an image quality difference on a boundary according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the configuration of a display device according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of an image quality difference notification process based on pixel class classification according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of pixel classification according to the second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a method of notifying a difference in image quality on a boundary according to the third embodiment, using a graph. [Figure 11]FIG. 11 is a diagram showing an example of a method of notifying a difference in image quality on a boundary according to the third embodiment, using a banner. DETAILED DESCRIPTION OF THE INVENTION

[0010] First Embodiment Hereinafter, an embodiment of the present invention will be described. An example will be described in which a display device has a built-in signal processing device according to the present embodiment, but the signal processing device according to the present embodiment may be a device (such as a personal computer) separate from the display device.

[0011] In this embodiment, the LED wall is a display device with LEDs arranged in a grid pattern. In in-camera VFX, performers and art sets are placed between the LED wall and the camera, background video is displayed on the LED wall, and the LED wall, performers, and art set are photographed together. Of the video photographed by in-camera VFX, the partial area originating from the performers and art set is called the real area, and the partial area originating from the LED wall is called the virtual area. Furthermore, when the virtual area plays the role of a background in video expression, it is called a background video.

[0012] Hereafter, differences in brightness and chromaticity are referred to as image quality differences. For example, if the brightness of the virtual area of ​​the captured image is high, the image quality difference can be eliminated by lowering the brightness of the background image displayed on the LED wall.

[0013] Fig. 1 is a block diagram showing an example of the configuration of a display device according to this embodiment. The display device 100 in Fig. 1 includes a receiving unit 101, a detecting unit 102, a control unit 103, a memory unit 104, an image processing unit 105, and a display unit 106.

[0014] The receiving unit 101 acquires video data and outputs the video data to the detecting unit 102. In this embodiment, the receiving unit 101 acquires frame data for each frame of the video from an external device. Then, the receiving unit 101 outputs the acquired frame data to the detecting unit 102. The receiving unit 101 acquires video data in accordance with a standard such as SDI or HDMI (registered trademark). The external device may be an imaging device or a playback device, and the receiving unit 101 may acquire video data recorded in the memory unit 104.

[0015] The detection unit 102 acquires the video data output from the receiving unit 101, detects region information from the metadata of the video data, and outputs the detected region information to the control unit 103. The metadata is, for example, ANC data standardized by SDI or data stored in InfoFrame standardized by HDMI. The metadata is identification information indicating for each pixel whether the pixel value was obtained by capturing an image of a real region or a virtual region. In this embodiment, this metadata is assigned to the video data by an external device. This identification information is determined, for example, based on a distance map obtained by a depth sensor. In this embodiment, region information is assigned by treating pixels obtained by capturing an image at a position farther away than a predetermined position as a virtual region and pixels obtained by capturing an image at a position closer to the predetermined position as a real region.

[0016] The control unit 103 controls the processing of each block of the display device 100. For example, the control unit 103 is an arithmetic processing circuit that executes a program stored in the memory unit 104 to control the processing of each block of the display device 100. In this embodiment, the control unit 103 controls the processing of the image processing unit 105 based on the region information and video data output from the detection unit 102.

[0017] The memory unit 104 stores programs, parameters, video data, etc. The programs, parameters, and video data stored in the memory unit 104 are called from each block of the display device 100.

[0018] The image processing unit 105 generates processed video data by performing image processing on the video data output from the detection unit 102 based on the control content output from the control unit 105. Then, the image processing unit 105 outputs the processed video data to the display unit 106.

[0019] Display unit 106 displays on its display surface a moving image based on the moving image data output from image processing unit 105. Display unit 106 is, for example, a liquid crystal display unit having a liquid crystal panel and a backlight unit, or an organic EL display panel.

[0020] 2 is a flowchart showing an example of a processing flow of the display device 100. When the frame image data acquired by the receiving unit 101 is updated, the processing flow of FIG.

[0021] In step S201, the detection unit 102 detects the coordinates of the boundary between the real area and the virtual area based on the area information acquired from the metadata, and notifies the control unit 103 of the detected coordinates.

[0022] 3 is a diagram showing an example of frame image data. A person 301 and a ground 302 are areas derived from the performers and the art set, and are objects in the real area. A tree 303 and a ground 304 are areas derived from the background image displayed on the LED wall, and are objects in the virtual area.

[0023] Fig. 4 is a diagram showing an example of a boundary detected from frame image data. In the metadata read from the video data, identification information indicating whether the pixel belongs to a real area or a virtual area is assigned to each pixel of the image by an external device. The detection unit 102 detects a portion where a pixel in a real area and a pixel in a virtual area are adjacent to each other as a boundary. In Fig. 4, the boundary 401 between the person 301 and the ground 304 and the boundary 402 between the ground 303 and the ground 304 are boundaries between real areas and virtual areas.

[0024] In step S202, the control unit 103 determines whether each coordinate of the boundary detected in step S201 is a boundary of the same object based on pixel values ​​around the coordinate. The pixel values ​​around each coordinate are the average values ​​of the luminance and chromaticity values ​​of pixels within a range of a predetermined distance (number of pixels) from the coordinate. FIG. 5 shows the area that is the target for averaging around a coordinate on the boundary. For coordinate 501, area 502 is the area that is the target for averaging pixel values ​​in the real area, and area 503 is the area that is the target for averaging pixel values ​​in the virtual area. In this embodiment, the average luminance Y1 and average chromaticity Cb1, Cr1 of the real area around the coordinate, and the average luminance Y2 and average chromaticity Cb2, Cr2 of the virtual area are used to determine whether the coordinate is a boundary of the same object using Equations 1 and 2. Equations 1 and 2 determine that the coordinate is a boundary of the same object if the average luminance difference and average chromaticity difference are less than predetermined thresholds α and β, respectively.

[0025]

number

[0026]

number

[0027] Then, the control unit 103 notifies the image processing unit 105 of the coordinates, luminance difference, and chromaticity difference determined to be the boundary of the same object. The notified boundary of the same object is, for example, the boundary 402 between the ground 303 and the ground 304 in FIG.

[0028] In step S203, the image processing unit 105 performs image processing to notify the user of the image quality difference based on the notified coordinates, luminance difference, and chromaticity difference. FIG. 6 shows an example of this notification. A guide display 601 is a legend indicating which colors are used depending on the magnitude of the luminance difference and chromaticity difference. A boundary line 602 is colored in a color corresponding to the magnitude of the luminance difference and chromaticity difference. Coloring refers to a process in which pixel values ​​of an image are changed so that they are perceived by the user as being replaced with a predetermined color or as being emphasized by a predetermined color component. Table 603 shows an example of the correspondence between legend patterns and colors. For example, a boundary where both the luminance difference and chromaticity difference are greater than a predetermined value is colored red. Note that a boundary where both the luminance difference and chromaticity difference are smaller than a predetermined value may not be colored. By coloring the boundary line in this way, it is possible to effectively indicate the extent of the image quality difference at which coordinate on the boundary line.

[0029] As described above, according to this embodiment, for the boundary between the real area and the virtual area detected from a video, it is determined whether the coordinates are the boundary of the same object based on the brightness and chromaticity around the boundary. If it is determined to be the boundary of the same object, the user is notified of the difference in image quality. Conventionally, the user had to identify the boundary between the real area and the virtual area in the captured video and determine whether there is a step due to a difference in image quality at the identified boundary. According to the present invention, if there is a difference in image quality at the boundary between the real area and the virtual area in the captured video, it is displayed in color, allowing the user to immediately determine the difference.

[0030] Note that the pixel values ​​of the video data may be converted into a color space closer to the perception of the human eye, such as the CIE 1976 (L*a*b*) color space, and then similar processing may be performed, which will enable detection closer to the perception of the human eye.

[0031] In addition, if the boundaries of different objects happen to have the same image quality, boundaries consisting of fewer pixels than a predetermined value may not be notified to prevent them from being notified even though they are not boundaries of the same object.

[0032] Note that each block in FIG. 1 of this embodiment may or may not be individual hardware. The functions of two or more blocks may be realized by common hardware. Each of multiple functions of one block may be realized by individual hardware. Two or more functions of one block may be realized by common hardware. Furthermore, each block may or may not be realized by hardware. For example, an apparatus may have a processor and a memory in which a control program is stored. Then, the functions of at least some of the blocks of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0033] Note that this embodiment (including the modified examples described below) is merely an example, and configurations obtained by appropriately modifying or changing the configuration of this embodiment within the scope of the gist of the present invention are also included in the present invention.

[0034] <Second embodiment> In the second embodiment, the display device classifies each pixel in the frame image data into a class (e.g., person, tree, ground, etc.). Then, based on the classification result and area information acquired from the metadata, the display device detects the coordinates that form the boundary between the real area and the virtual area. The display device notifies the user of the image quality difference at the detected coordinates.

[0035] Fig. 7 is a block diagram showing an example of the configuration of a display device according to the second embodiment. The display device 700 in Fig. 7 has an identification unit 701 in addition to the same blocks as in the first embodiment. Note that the same reference numerals are used for the same blocks as in the first embodiment, and their explanations will be omitted. The control unit 702 operates differently from the first embodiment, and is therefore assigned a different reference numeral.

[0036] The identification unit 701 acquires the video data output from the receiving unit 101, classifies each pixel in the frame image data of the video data into a class (for example, person, tree, ground, etc.), and outputs the classification result to the control unit 702. The classification method is, for example, semantic segmentation.

[0037] The control unit 702 controls the processing of the image processing unit 105 based on the region information output from the detection unit 102, the classification result output from the identification unit 701, and the video data.

[0038] 8 is a flowchart showing an example of a processing flow of the display device 700. Note that the same processes as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0039] In step S801, the classification unit 701 classifies each pixel in the frame image data of the video data into a class. Fig. 9 is a diagram showing an example of the classification of each pixel by the classification unit. Each class is painted with a different pattern, with class 901 representing people, class 902 representing trees, and class 903 representing the ground.

[0040] In step S802, the control unit 702 determines whether each coordinate of the boundary detected in step S201 is a boundary of the same object based on the class of the pixels around the coordinate, which were classified in step S801. The class of the pixels around the coordinate is the class that occupies the largest proportion within a predetermined range on both the real area side and the virtual area side. If the class on the real area side and the class on the virtual area side are the same for the pixel on the boundary line, it is determined to be a boundary of the same object.

[0041] As described above, according to this embodiment, for a boundary between a real area and a virtual area detected from a video, it is determined whether the coordinates are the boundary of the same object based on the class around the boundary. If it is determined that the coordinates are the boundary of the same object, the user is notified of the difference in image quality. According to the present invention, if there is a difference in image quality at the boundary between the real area and the virtual area of ​​the captured video, it is displayed in color, allowing the user to immediately determine that there is a difference in brightness or chromaticity at the boundary.

[0042] <Third embodiment> In the third embodiment, a method for notifying the difference in image quality at the boundary of the same object that has been detected will be described, other than the color display method described in the first embodiment.

[0043] 10 is a diagram showing an example of notifying image quality differences using a brightness graph. Graph 1001 shows the brightness difference at the x-coordinate of the boundary of the same object. The brightness value on the vertical axis of the graph is calculated using Equation 3, using the average brightness Y1 of the real area around each coordinate on the boundary and the average brightness Y2 of the virtual area around that coordinate.

[0044]

number

[0045] If there is no boundary at the x-coordinate, the brightness difference corresponding to that x will not be plotted on the graph. If there are two or more boundaries at the same x-coordinate, the brightness difference corresponding to that x will be plotted as many times as the number of boundaries.

[0046] Users can eliminate image quality differences at the boundaries by adjusting the brightness of the background image so that the brightness difference graph remains within a specified range overall. The graph notification method has the advantage of showing the degree of image quality difference at each coordinate, allowing users to fine-tune the image quality difference at each coordinate.

[0047] FIG. 11 shows an example of notifying image quality differences using a banner. Banner 1101 shows the average difference in brightness at the boundary of the same object. The banner displays the type and magnitude of the image quality difference. The numerical value notified in the banner is calculated using Equation 4 by taking the average brightness difference at each coordinate handled in the brightness graph described above.

[0048]

number

[0049] The user simply adjusts the brightness of the camera so that the notified brightness difference approaches 0, and can make adjustments while understanding the brightness discrepancy in real time. In this way, the banner notification method has the advantage of making it easy for the user to understand what adjustments need to be made and by how much to achieve the desired image quality.

[0050] Although the method of notifying the luminance Y has been described above, the same method can be used for notifying the chromaticity Cx and Cy.

[0051] As described above, according to this embodiment, by using an appropriate display method to notify the user of differences in brightness or chromaticity at the boundary of the same object, the user can immediately determine that there is a difference in brightness or chromaticity at the boundary. Furthermore, the user can intuitively understand the positions where image quality adjustment is required, and can easily adjust the image quality difference for each coordinate.

[0052] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0053] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0054] In the above embodiment, at least one of A and B may be only A, only B, or both A and B.

[0055] The disclosure of this embodiment includes the following configuration.

[0056] (Configuration 1) receiving means for receiving image data; a detection means for detecting a first image area corresponding to a virtual area and a second image area corresponding to a real area from the image data; a display means for displaying a boundary between the first area and the second area, the boundary being an area where both the first area and the second area indicate the same object; A signal processing device comprising:

[0057] (Configuration 2) The display means displays a boundary between the first region and the second region, where both the first region and the second region indicate the same object, based on at least one of luminance and chromaticity of pixels around the boundary between the first region and the second region. 2. The signal processing device according to configuration 1,

[0058] (Configuration 3) 3. The signal processing device according to configuration 1 or 2, wherein the notification means notifies an area where the difference in at least one of the luminance and chromaticity of pixels around the boundary is small.

[0059] (Configuration 4) 4. The signal processing device according to any one of configurations 1 to 3, wherein the notification means displays each boundary coordinate in a different color depending on the magnitude of the difference in image quality.

[0060] (Configuration 5) 5. The signal processing device according to any one of configurations 1 to 4, wherein the notification means displays the coordinates of the image and the magnitude of the image quality difference as a graph.

[0061] (Configuration 6) 6. The signal processing device according to any one of configurations 1 to 5, wherein the notification means displays the type and magnitude of the image quality difference.

Claims

1. receiving means for receiving image data; a detection means for detecting a first region corresponding to a virtual region and a second region corresponding to a real region from the image data; a display means for displaying a boundary between the first area and the second area, the boundary being an area where both the first area and the second area indicate the same object; A signal processing device comprising:

2. The display means displays a boundary between the first region and the second region, the boundary being a region where both the first region and the second region indicate the same object, based on at least one of luminance and chromaticity of pixels around the boundary between the first region and the second region.

2. The signal processing device according to claim 1.

3. 2. The signal processing apparatus according to claim 1, wherein said display means displays an area where the difference in at least one of brightness and chromaticity of pixels around the boundary is small.

4. 2. The signal processing device according to claim 1, wherein said display means displays each boundary coordinate in a different color depending on the magnitude of the difference in image quality.

5. 2. The signal processing apparatus according to claim 1, wherein said display means displays the coordinates of the image and the magnitude of the image quality difference as a graph.

6. 2. The signal processing apparatus according to claim 1, wherein said display means displays the type and magnitude of the image quality difference.

7. a receiving step of receiving image data; a detection step of detecting a first region corresponding to a virtual region and a second region corresponding to a real region from the image data; a display step of displaying a boundary between the first area and the second area, the boundary being an area where both the first area and the second area indicate the same object; A method for controlling a signal processing device having the above-mentioned components.

8. a receiving step of receiving image data; a detection step of detecting a first region corresponding to a virtual region and a second region corresponding to a real region from the image data; a display step of displaying a boundary between the first region and the second region, the boundary being a region where both the first region and the second region indicate the same object; A program that causes a computer to execute the following.

Images