Display system, information processing system, and program

Abstract

The present technology relates to a display system, an information processing system, and a program that enable enhancement of the image quality of captured video when a subject is captured against the background of a video being displayed on a display. A display system according to one aspect of the present technology displays, on a display used for displaying a background video, a display video having a frame configuration in which a background image, which is a frame image constituting the background video captured as the background of a subject, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color having a lower saturation than the predetermined color, are arranged in a predetermined order. The present technology can be applied to video production using virtual production imaging.

Description

Display systems, information processing systems, programs

[0001] The present technology relates to a display system, an information processing system, and a program, and in particular to a display system, an information processing system, and a program that enable high image quality to be achieved when photographing a subject with an image displayed on a display as the background.

[0002] Virtual production, a filming technique that uses large LED (Light Emitting Diode) displays, is becoming increasingly popular in the filming of movies and TV dramas. Filming using virtual production involves filming subjects, such as actors, performing in front of an LED display that displays a background image.

[0003] By using virtual production, it becomes possible to shoot movies and other productions without being restricted by weather, time, location, etc.

[0004] International Publication No. 2023 / 047643

[0005] If there is any reflection of lighting or other objects on the LED display that displays the background image, that reflection will also be captured as a reflected image, making the background in the captured image unrealistic. This can occur especially when the background image is dark.

[0006] Furthermore, if the LED display has uneven reflections, these will also be captured as reflected images, reducing the contrast of the background.

[0007] The present technology has been developed in consideration of such circumstances, and makes it possible to achieve high image quality in captured images when capturing a subject with an image displayed on a display as the background.

[0008] A display system according to one aspect of the present technology includes a display control unit that causes a display image having a frame configuration in which a background image, which is a frame image constituting a background image photographed as the background of a subject, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order on a display used to display the background image.

[0009] An information processing system according to another aspect of the present technology includes an acquisition unit that acquires a captured image obtained by capturing a subject against a background of a display that displays a display image having a frame configuration in which a background image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order, and an image processing unit that detects an area of ​​the subject based on a first captured image captured in synchronization with the display timing of the first color image, removes the area of ​​the subject from a second captured image captured in synchronization with the display timing of the second color image, and corrects the background captured image captured in synchronization with the display timing of the background image based on the second captured image from which the area of ​​the subject has been removed.

[0010] In one aspect of the present technology, a display image having a frame configuration in which a background image, which is a frame image constituting a background image captured as the background of a subject, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order, is displayed on a display used to display the background image.

[0011] In another aspect of the present technology, a captured image is obtained by capturing an image of a subject against a display that displays an image having a frame configuration in which a background image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order, the area of ​​the subject is detected based on the first captured image captured in synchronization with the display timing of the first color image, the area of ​​the subject is removed from the second captured image captured in synchronization with the display timing of the second color image, and the background captured image captured in synchronization with the display timing of the background image is corrected based on the second captured image from which the area of ​​the subject has been removed.

[0012] 1 is a diagram showing how shooting is performed using a shooting system according to an embodiment of the present technology. FIG. 1 is a diagram showing an example of the configuration of the shooting system. FIG. 2 is a diagram showing the flow of video production using the shooting system. FIG. 3 is a diagram showing an example of uneven reflection in virtual production shooting. FIG. 4 is a diagram showing an example of reflection in virtual production shooting. FIG. 5 is a diagram showing an example of a series of processing flows for each device. FIG. 6 is a diagram showing an example of three types of video. FIG. 7 is a diagram showing an example of synchronization between devices. FIG. 8 is a diagram showing an example of signal processing 1 when uneven reflection is present. FIG. 9 is a diagram showing an example of signal processing 2 when reflection is present. FIG. 10 is a block diagram showing an example of the hardware configuration of a PC. FIG. 11 is a block diagram showing an example of the functional configuration of a video server and a display controller. FIG. 12 is a block diagram showing an example of the functional configuration of an image processing device. FIG. 13 is a flowchart showing the operation of a display system. FIG. 14 is a flowchart showing the operation of an image processing device. FIG. 15 is a diagram showing an example of a frame frequency. FIG. 16 is a diagram showing another example of frame configurations of three types of video. FIG. 17 is a diagram showing an example of moiré in shot video. FIG. 18 is a diagram explaining the principle of moiré generation. FIG. 19 is a diagram showing an example of a series of processing flows for each device. FIG. 19 is a diagram showing an example of three types of video used for moiré removal. FIG. 19 is a diagram showing an example of signal processing 1 when moiré is present. FIG. 19 is a diagram showing an example of signal processing 2 when moiré is present.

[0013] Hereinafter, embodiments of the present technology will be described in the following order: 1. Virtual production photography 2. Removal of reflection unevenness and glare 3. Configuration and operation of each device 4. Modified example

[0014] <<Virtual Production Shooting>> <Regarding the Shooting System> FIG. 1 is a diagram showing a state of shooting using a shooting system according to an embodiment of the present technology.

[0015] 1 is provided with an LED display 11 and a video camera 12. The photography system 1 is also provided with other devices such as a controller for video output, a monitor for checking the video, multiple video cameras other than the video camera 12, and lighting equipment.

[0016] The imaging system 1 is prepared in a large space such as a photography studio. Using the imaging system 1, video content such as movies and dramas is shot by so-called virtual production shooting.

[0017] A large wall-shaped LED display 11 is installed in the photography studio. As will be described later, the LED display 11 is configured by combining multiple display units in a tiled pattern. A large image is displayed on the entire display surface of the LED display 11 by combining the images displayed by the multiple display units. In the example of FIG. 1, an image of a city night view is displayed.

[0018] 1, subject H, who is a performer, stands in front of an LED display 11 and performs with the image displayed on the LED display 11 as the background. The image displayed on the LED display 11 becomes the background image reflected on the background of subject H.

[0019] Virtual production shooting is performed by shooting subject H and background video with video camera 12. The video shot by video camera 12 includes at least a portion of the range of both subject H and background video, depending on the shooting range (angle of view) of video camera 12. By changing the background video depending on the movement of subject H and the movement of video camera 12, a video is shot that makes it appear as if subject H is in the space projected by LED display 11.

[0020] FIG. 2 is a diagram showing an example of the configuration of the imaging system 1.

[0021] As shown in FIG. 2, the photographing system 1 includes an LED display 11 and a video camera 12, as well as a video server 31 and a plurality of display controllers 32.

[0022] The LED display 11 is configured by arranging a plurality of display units 21, 21-1 to 21-n, in a tiled arrangement. As shown in the speech bubble, each display unit 21 is configured by arranging an LED array in a tiled arrangement, with an LED corresponding to each pixel. In the example of Figure 2, one display unit 21 is configured by a 4 x 3 LED array.

[0023] The video camera 12 captures images in response to the cameraman's operation or under the control of a computer (not shown). A captured video signal representing the captured video is output to an editing PC. The captured video signal is output to the editing PC via wired or wireless communication or by using removable media such as an SSD (Solid State Drive).

[0024] The video server 31 is configured by a computer such as a PC. A background video signal, which is a signal for a background video created, for example, on a content production PC, is input to the video server 31. The background video signal is input via wired or wireless communication or using removable media. The video server 31 assigns a video to be displayed on each display unit 21 and outputs the video signal for each assigned frame to each display controller 32.

[0025] The display controller 32 is a device that controls the display of the display units 21 that make up the LED display 11. One or more display units 21 are connected to one display controller 32. The display controller 32 outputs a background video signal to the display units 21, causing them to display the background video of each frame. The display of each display unit 21 is controlled by the respective display controller 32, so that one frame of the background video is displayed on the entire display surface of the LED display 11.

[0026] In this way, the LED display 11, the video server 31, and the display controller 32 constitute a display system in the imaging system 1. The display system is configured to include the display controller 32, which is a device equipped with a function for controlling the display of the LED display 11.

[0027] FIG. 3 is a diagram showing the flow of video production using the shooting system 1.

[0028] Video production using the shooting system 1 is mainly divided into pre-production, shooting, and post-production work.

[0029] As shown on the left side of FIG. 3, as a pre-production step, background images such as the above-mentioned night view images are produced.

[0030] Following the production of the background video, shooting is performed using the shooting system 1. As indicated by the end of arrow A1, in shooting using the shooting system 1, three types of video including the background video are displayed on the LED display 11, and as indicated by the end of arrow A2, shooting is performed by the video camera 12 in synchronization with the display of the three types of video. The three types of video are display images used for display on the LED display 11. Details of the three types of video and shooting synchronized with the display of the three types of video will be described later.

[0031] The video captured by the video camera 12 is input into an editing PC as indicated by the tip of arrow A3, and undergoes signal processing 1 and signal processing 2 as post-production work. Signal processing 1 and signal processing 2 remove uneven reflections and glare contained in the captured video. Details of signal processing 1 and signal processing 2 will be described later.

[0032] The captured video, from which reflection irregularities and glare have been removed by signal processing 1 and signal processing 2, is used for video editing, as indicated by the arrow A4. Signal processing 1 and signal processing 2 correspond to pre-processing for generating video for editing. In video editing, other materials such as CG are also used as appropriate, as indicated by arrow A5. After video editing, audio editing is performed, as indicated by arrow A6, to complete the video content.

[0033] In this way, in video production using the imaging system 1, the three types of images are displayed on the LED display 11 to remove uneven reflections and glare from the captured video, and the video is shot in synchronization with the display of the three types of images. In addition, in post-production, signal processing is performed on the captured video to remove uneven reflections and glare.

[0034] <<Removal of Uneven Reflection and Reflection>> <Examples of Uneven Reflection and Reflection> FIG. 4 is a diagram showing an example of uneven reflection in virtual production shooting.

[0035] The LED display in Figure 4 shows an image of a city nightscape as the background image. When displaying a dark image such as a nightscape image, uneven reflections such as uneven joints and uneven tile surfaces can occur depending on the intensity and angle of the lighting directed at the subject, as shown in the speech bubble. As mentioned above, an LED display is made up of multiple display units combined in a tiled pattern, and uneven tile and uneven joint surfaces appear at the joints of the display units.

[0036] If a photo is taken with uneven reflections, the uneven reflections will be captured as a reflected image, as shown by the white arrow. In the captured image, the black parts of the night scene will appear whitish, reducing the contrast.

[0037] FIG. 5 is a diagram showing an example of reflections in virtual production shooting.

[0038] The LED display in Figure 5 also displays a city night view as a background image. When displaying dark images such as night views, depending on the strength and angle of the lighting directed at the subject, the lighting may be reflected, as shown in the speech bubble.

[0039] If a photo is taken with a glare present, the glare will be captured as a reflected image, as shown by the white arrow, and will be included in the captured video as is.

[0040] As described above, in the imaging system 1, uneven reflection and glare may occur, particularly when a dark image is displayed as the background image.

[0041] <Processing of each device in video production> Fig. 6 is a diagram showing an example of a series of processing flows of each device in video production using the shooting system 1. Fig. 6 shows the processing during shooting and the processing during post-production that were described with reference to Fig. 3.

[0042] 6, when shooting using the shooting system 1, the video server 31 generates three types of images including a background image and outputs signals of the three types of images to the display controller 32. The display controller 32 displays the three types of images on the LED display 11 based on the signals input from the video server 31.

[0043] FIG. 7 shows examples of three types of images that are displayed on the LED display 11. In FIG.

[0044] As shown in Fig. 7, the three-type video has a frame configuration in which frames of a background image, a black image, and a monochrome image are arranged in this order. For each frame of the background image, one frame of a black image and one frame of a monochrome image are added. The three-type video shown in Fig. 7 has a frame configuration in which one frame of a black image and one frame of a monochrome image are arranged between frames of a background image. The display order (arrangement order) of one set of three types of frames of a background image, a black image, and a monochrome image can be changed as desired.

[0045] A background image is an image of one frame (frame image) of a background video. A background video is made up of multiple frames of background images.

[0046] A black image is an image of black color. The pixel values ​​of each pixel of a black image are, for example, RGB = 0,0,0. Instead of black, an image of another color with low brightness, such as dark gray, may be used.

[0047] A monochrome image is an image of a predetermined color with high saturation, such as green, blue, red, yellow, cyan, magenta, etc. The monochrome image, which is the first color image, is an image with higher saturation than the black image, which is the second color image. In other words, the black image is an image with lower saturation than the monochrome image.

[0048] If the frame frequency of the background image is 120 Hz, three frames of the background image, black image, and monochrome image are displayed within a 120 Hz period, as indicated by the two-way arrow in FIG.

[0049] The video camera 12 takes a picture of the subject with the three types of images displayed in the background in synchronization with the display of these three types of images.

[0050] FIG. 8 is a diagram illustrating an example of synchronization between devices.

[0051] A clock generator 33 is provided at a predetermined position in the imaging system 1. The clock generator 33 and the video camera 12, the clock generator 33 and the video server 31, and the clock generator 33 and the display controller 32 are connected by cables, respectively. The clock generator 33 outputs a synchronization pulse signal (Gen Lock signal) to each of the video camera 12, the video server 31, and the display controller 32 to achieve frame synchronization. For example, the shooting timing (exposure start time) of each frame in the video camera 12 is synchronized with the display timing (display start time) of each frame on the LED display 11.

[0052] Each frame is photographed in accordance with a synchronization pulse signal, and a photographed video consisting of an image photographed in synchronization with the display timing of a background image, an image photographed in synchronization with the display timing of a black image, and an image photographed in synchronization with the display timing of a monochrome image is recorded.

[0053] Among the captured images, an image captured in synchronization with the display timing of a background image is called a background captured image, an image captured in synchronization with the display timing of a black image is called a black-image captured image, and an image captured in synchronization with the display timing of a monochrome image is called a monochrome captured image.

[0054] A background photographed image is an image in which a background image is projected on the background of a subject. A black-image photographed image is an image in which a black image is projected on the background of a subject. A monochrome photographed image is an image in which a monochrome image is projected on the background of a subject. A photographed video consisting of the background photographed image, black-image photographed image, and monochrome photographed image is taken into the image processing device 51 as shown in the lower part of Figure 6.

[0055] The image processing device 51 is an editing PC used in post-production work. The shot video captured by the image processing device 51 includes a set of background shot images, black-screen shot images, and monochrome shot images. Of the background shot images, black-screen shot images, and monochrome shot images, the black-screen shot images and monochrome shot images are used in signal processing 1. The background shot images and the black-screen shot images from which the subject has been removed, generated by signal processing 1, are used in signal processing 2. Signal processing 1 and signal processing 2 are performed using the background shot images, black-screen shot images, and monochrome shot images that constitute the same set. The image processing device 51 realizes an information processing system that processes the background shot images, black-screen shot images, and monochrome shot images.

[0056] Signal Processing 1 (Case with Uneven Reflection) FIG. 9 is a diagram showing an example of signal processing 1 when uneven reflection exists.

[0057] As shown on the left side of Figure 9, the black image is an image in which a black image is projected onto the background of the subject, and reflection irregularities are projected onto the black portion on the left side of the subject.

[0058] In signal processing 1, a subject area is detected from a monochromatic captured image by chromakey processing using the monochromatic captured image. As shown in the center of Fig. 9, the monochromatic captured image is an image in which a highly saturated color such as green appears in the background of the subject. The hatched areas in the monochromatic captured image in Fig. 9 are green areas. The subject area is detected by detecting areas other than the green area.

[0059] By removing the subject area detected by the chromakey process from the black-screen image, a subject-removed black-screen image is generated, as indicated by the white arrow. In the subject-removed black-screen image shown on the right side of Figure 9, the hatched subject area indicates that the subject area has been removed. The subject-removed black-screen image generated in this manner is used in signal processing 2.

[0060] Signal Processing 2 (Case with Uneven Reflection) FIG. 10 is a diagram showing an example of signal processing 2 when uneven reflection exists.

[0061] As shown on the left side of Fig. 10, the background image is an image in which a background image such as a city night view is reflected in the background of the subject. Reflection irregularities are reflected in the black area on the left side of the subject.

[0062] In signal processing 2, a signal-processed captured image is generated, as indicated by the white arrow, by, for example, subtracting the subject-removed black image from the background captured image. The signal-processed captured image is generated by subtracting the pixel value (signal level) of each pixel in the subject-removed black image from the pixel value (signal level) of the corresponding pixel in the background captured image. The signal-processed captured image may also be generated by calculations other than subtraction processing using the background captured image and the subject-removed black image.

[0063] The subject-removed black image generated by signal processing 1 is an image that mainly contains components of uneven reflection. By subtracting the components of uneven reflection from the background image, the signal-processed image is generated as a background image from which the components of uneven reflection have been removed, as shown on the right side of Figure 10. The signal-processed image has a higher contrast than the background image. Reflections contained in the image are also removed by processing similar to that described above.

[0064] Signal Processing 1 (Case where Reflection is Present) Fig. 11 is a diagram showing an example of signal processing 1 when reflection is present. Description that overlaps with the above description will be omitted as appropriate.

[0065] As shown on the left side of Fig. 11, the black image is an image in which a black image is projected onto the background of the subject, and the black portion on the left side of the subject contains reflections of lighting and the like.

[0066] In signal processing 1, a subject area is detected from the monochrome photographed image by chromakey processing using the monochrome photographed image, and the subject area detected by chromakey processing is removed from the black photographed image to generate a subject-removed black photographed image, as indicated by the white arrow.

[0067] Signal Processing 2 (Case with Reflection) FIG. 12 is a diagram showing an example of signal processing 2 when reflection is present.

[0068] In signal processing 2, a signal-processed image is generated by subtracting the subject-removed black image from the background image, as indicated by the white arrow. The signal-processed image is generated by subtracting the pixel value of each pixel in the subject-removed black image from the pixel value of the corresponding pixel in the background image.

[0069] The subject-removed black image generated by signal processing 1 is an image that mainly contains glare components. By subtracting the glare components from the background image, the signal-processed image is generated as a background image from which the glare components have been removed, as shown on the right side of Fig. 12. The signal-processed image has a higher contrast than the background image.

[0070] By performing the above-described signal processing in the image processing device 51, it is possible to remove uneven reflections and glare even when they are included in the captured image captured using the imaging system 1. The captured image with high contrast and high image quality can be used for video editing (FIG. 3), making it possible to produce high-quality video content.

[0071] <<Configuration and Operation of Each Device>> <Configuration of Each Device> Hardware Configuration Fig. 13 is a block diagram showing an example of the hardware configuration of the PC 101. The video server 31, display controller 32, and image processing device 51 are configured by a computer having the configuration shown in Fig. 13. In the following description, the configuration shown in Fig. 13 will be cited as the configuration of the video server 31, display controller 32, and image processing device 51 where appropriate.

[0072] In the PC 101 of FIG. 13, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113 are interconnected by a bus 114.

[0073] An input / output interface 115 is connected to the bus 114. An input unit 116 including a keyboard, a mouse, etc., and an output unit 117 including a display, a speaker, etc. are connected to the input / output interface 115.

[0074] Furthermore, the input / output interface 115 is connected to a storage unit 118 such as a hard disk or nonvolatile memory, a communication unit 119 such as a network interface, and a drive 120 that drives a removable medium 121 .

[0075] The communication unit 119 communicates with each device. For example, the communication unit 119 of the image processing device 51 receives camera information including video images captured by the video camera 12 and shooting parameters such as the position and shooting range of the video camera 12. This information is transmitted, for example, from the video camera 12 used for shooting or from a device that controls the operation of the video camera 12 used for shooting.

[0076] Functional Configuration FIG. 14 is a block diagram showing an example of the functional configuration of the video server 31 and the display controller 32 that constitute the display system.

[0077] 14, a background image acquisition unit 201, a three-type image generation unit 202, and a three-type image output unit 203 are realized in the video server 31, and a display control unit 211 is realized in the display controller 32. Each functional unit is realized by the CPU 111 constituting the device executing a predetermined program.

[0078] The background video acquisition unit 201 of the video server 31 receives a signal input from a content production PC used in pre-production and acquires background video. The background video acquired by the background video acquisition unit 201 is output to the three-video generation unit 202.

[0079] The three-type video generation unit 202 generates three-type videos by adding a black image frame and a monochrome image frame to each frame of the background video supplied from the background video acquisition unit 201. The three-type videos generated by the three-type video generation unit 202 are output to the three-type video output unit 203.

[0080] The three-type video output unit 203 outputs signals for each frame of the three types of video to the display controller 32 in accordance with a synchronization pulse signal supplied from a clock generator 33 (not shown in FIG. 14).

[0081] The display control section 211 of the display controller 32 receives the signal input from the video server 31 and displays each frame of the three types of video on the LED display 11 in accordance with the synchronization pulse signal supplied from the clock generator 33. The display of the three types of video on the LED display 11 is synchronized with the timing of the video camera 12 capturing the subject.

[0082] FIG. 15 is a block diagram showing an example of the configuration of the image processing device 51.

[0083] As shown in FIG. 15, an image processing device 51 includes a captured image acquisition unit 251 and an image processing unit 252 .

[0084] The shot video acquisition unit 251 acquires the shot video that is input into the image processing device 51 during post-production work. The shot video acquired by the shot video acquisition unit 251 is output to the image processing unit 252.

[0085] The image processing unit 252 performs signal processing 1 based on the black-image captured background image and the monochrome captured image among the background captured background image, the black-image captured black image, and the monochrome captured image included in the captured image supplied from the captured image acquisition unit 251. The image processing unit 252 also performs signal processing 2 based on the background captured background image and the subject-removed black-image captured black image generated by signal processing 1, thereby removing uneven reflections and glare included in the background captured image. In this way, the image processing unit 252 has the function of detecting the subject area based on the monochrome captured image as the first captured image, removing the subject area from the black-image captured black image as the second captured image, and correcting the background captured image based on the black-image captured black image from which the subject area has been removed. The correction of the background captured image is performed so as to remove the reflected image included in the background captured image.

[0086] The signal-processed captured image generated by the image processing unit 252 is used for video editing in an editing processing unit (not shown). The video editing may be performed in the same PC as the image processing device 51, or may be performed in a PC separate from the image processing device 51.

[0087] <Operation of Each Device> The operation of the video server 31 and the display controller 32 that constitute the display system will be described with reference to the flowchart in Fig. 16. The process in Fig. 16 starts when, for example, a background video signal is input to the video server 31.

[0088] In step S1, the background image acquisition unit 201 of the video server 31 receives and acquires the background image input from the content production PC.

[0089] In step S2, the three-type video generating unit 202 of the video server 31 adds a black image frame and a monochrome image frame to each frame of the background video to generate three types of video.

[0090] In step S 3 , the three-type video output unit 203 of the video server 31 outputs the three-type video signals to the display controller 32 in accordance with the synchronization pulse signal supplied from the clock generator 33 .

[0091] In step S4, the display control section 211 of the display controller 32 receives the signal supplied from the video server 31 and displays each frame of the three types of video on the LED display 11 in accordance with the synchronization pulse signal supplied from the clock generator 33. The processing of Fig. 16 continues while shooting by the video camera 12 is being performed.

[0092] Next, the operation of the image processing device 51 will be described with reference to the flowchart of Fig. 17. The process of Fig. 17 starts, for example, when the video camera 12 finishes shooting and the shot video is input.

[0093] In step S11, the captured video acquisition unit 251 of the image processing device 51 acquires a captured video.

[0094] In step S12, the image processing unit 252 detects a subject area based on the monochrome captured image from among the background captured image, the black image captured image, and the monochrome captured image included in the captured video.

[0095] In step S13, the image processing unit 252 removes the subject area from the captured black image to generate a subject-removed captured black image. The processes in steps S12 and S13 correspond to signal processing 1.

[0096] In step S14, the image processing unit 252 subtracts the subject-removed black image from the background image to generate a signal-processed image. The process in step S14 corresponds to signal processing 2.

[0097] The processes of steps S12 to S14 are repeated for each set of background captured images, black captured images, and monochrome captured images that make up the captured video. This makes it possible to obtain high-contrast captured images from which reflection irregularities and glare contained in the captured video have been removed, thereby achieving high-quality captured video.

[0098] <<Modifications>> <Modifications of Frame Frequency> FIG. 18 is a diagram showing another example of the frame frequency of the background image.

[0099] The frame frequency of the background image shown in Fig. 18 is 24 Hz. When three types of video are constructed by sequentially arranging one frame each of the background image, black image, and monochrome image, the frequency of the synchronization pulse signal is 72 Hz, as shown in Fig. 18.

[0100] In this way, any frequency such as 24 Hz, 23.98 Hz, 48 Hz, 47.96 Hz, 59.94 Hz, 60 Hz, or 120 Hz can be used as the frame frequency of the background image.

[0101] <Modification of Frame Structure> FIG. 19 is a diagram showing another example of a frame structure of three types of video.

[0102] One set of three types of video shown in Figure 19 is composed of three frames of background image, one frame of black image, and one frame of monochrome image. The three types of video are composed of three frames of background image followed by one frame each of black image and monochrome image. As indicated by the double arrow, the frame frequency of each frame is 120 Hz.

[0103] 19, background images P1-1, P1-2, P1-3, black image P2, and monochrome image P3 form one set of frames for the three types of video. Also, background images P4-1, P4-2, P4-3, black image P5, and monochrome image P6 form one set of frames for the three types of video.

[0104] In post-production, signal processing 1 is performed using a black-image photographed in synchronization with the display timing of black image P2 and a monochrome photographed image photographed in synchronization with the display timing of monochrome image P3, to generate a subject-removed black-image photographed. Furthermore, signal processing 2 using the subject-removed black-image photographed image removes uneven reflections and glare from the background photographed images photographed in synchronization with the display timing of each of background images P1-1, P1-2, and P1-3.

[0105] Similarly, signal processing 1 is performed using a black-image photographed in synchronization with the display timing of black image P5 and a monochrome photographed image in synchronization with the display timing of monochrome image P6, to generate a subject-removed black-image photographed. Furthermore, signal processing 2 using the subject-removed black-image photographed image removes uneven reflections and glare from the background photographed images photographed in synchronization with the display timing of each of background images P4-1, P4-2, and P4-3.

[0106] In this way, a configuration in which multiple frames of a background image are arranged as a frame configuration for one set of three types of video can be used. In addition to background images, multiple frames of black images and monochrome images may also be arranged in one set of three types of video. One set of three types of video may include more frames of black images and monochrome images than frames of background images.

[0107] <Moire Removal> Principle of Moire Generation FIG. 20 is a diagram showing an example of moire on a captured image.

[0108] In virtual production shooting, where a background image displayed on an LED display is shot together with the subject, moiré can occur in the shot image, as shown in the bottom part of Figure 20. Unlike uneven reflections or glare, moiré is more noticeable when the background image is bright. Moiré also appears throughout the entire shot image.

[0109] As shown in Figure 21, moiré appears when a video is shot with a slight misalignment between the pixel array of the video camera's imaging element and the pixel array of the LED display used to display the background image. The misalignment of the pixel arrays causes moiré, degrading the quality of the shot image. Moiré is an interference component in the shot image.

[0110] The above-described technology can also be applied to removing moiré. In this case, in video production using the shooting system 1, three types of images are displayed on the LED display 11 to remove moiré from the shot video, and shooting is performed in synchronization with the display of the three types of images. In addition, in post-production, signal processing to remove moiré is performed on the shot video.

[0111] Processing of each device in video production Fig. 22 is a diagram showing an example of the flow of a series of processes of each device in video production using the shooting system 1. Descriptions that overlap with the content explained with reference to Fig. 6 will be omitted as appropriate.

[0112] 22, when shooting using the shooting system 1, the video server 31 generates three types of images including a background image and outputs signals of the three types of images to the display controller 32. The display controller 32 displays the three types of images on the LED display 11 based on the signals input from the video server 31.

[0113] FIG. 23 shows examples of three types of images used for moire removal.

[0114] As shown in Fig. 23, the three types of images used for moire removal have a frame structure in which frames of a background image, a gray image, and a monochrome image are arranged in this order. For each frame of the background image, one frame of a gray image and one frame of a monochrome image are added. The display order of the three types of images, the background image, the gray image, and the monochrome image, can be changed as desired.

[0115] In this way, in moiré removal, a gray image, which is also an achromatic image, is used instead of the black image used to remove reflection unevenness and glare. The background image and the monochrome image are the same as the background image and monochrome image of the three-color video used to remove reflection unevenness and glare. In this case, too, the monochrome image, which is the first color image, has a higher saturation than the gray image, which is the second color image. In other words, the gray image has a lower saturation than the monochrome image.

[0116] Instead of a gray image, a white image may be used. The pixel values ​​of each pixel in the white image are, for example, RGB = 255, 255, 255. The white image is also an achromatic image, just like the black image and gray image.

[0117] If the frame frequency of the background image is 120 Hz, three frames of the background image, gray image, and monochrome image are displayed within a 120 Hz period, as shown by the two-way arrow in FIG.

[0118] The video camera 12 takes a picture of the subject with the three types of images displayed in the background in synchronization with the display of these three types of images.

[0119] Each frame is photographed in accordance with a synchronization pulse signal, and a photographed video consisting of an image photographed in synchronization with the display timing of a background image, an image photographed in synchronization with the display timing of a gray image, and an image photographed in synchronization with the display timing of a monochrome image is recorded.

[0120] An image captured in synchronization with the display timing of the gray image is called a gray image captured image. A gray image captured image is an image in which a gray image is projected onto the background of a subject. A captured video consisting of the background captured image, gray image captured image, and monochrome captured image is input to the image processing device 51 as shown in the lower part of Figure 22.

[0121] Of the background photographed image, grayscale photographed image, and monochrome photographed image contained in the photographed video captured in the image processing device 51, the grayscale photographed image and the monochrome photographed image are used in signal processing 1. Furthermore, the background photographed image and the grayscale photographed image from which the subject has been removed, generated by signal processing 1, are used in signal processing 2. Signal processing 1 and signal processing 2 are performed using the background photographed image, grayscale photographed image, and monochrome photographed image that constitute the same set. The image processing device 51 realizes an information processing system that processes the background photographed image, grayscale photographed image, and monochrome photographed image.

[0122] Signal Processing 1 (Case where Moire is Present) FIG. 24 is a diagram showing an example of signal processing 1 when moire is present.

[0123] As shown on the left side of Fig. 24, the grayscale photographed image is an image in which a grayscale image is projected onto the background of the subject. Moire is projected onto the entire grayscale photographed image.

[0124] In signal processing 1, a subject area is detected from a monochromatic captured image by chromakey processing using the monochromatic captured image. As shown in the center of Fig. 24, a monochromatic captured image is an image in which a highly saturated color such as green appears in the background of the subject. The subject area is detected by detecting areas other than the green area.

[0125] By removing the subject area detected by the chromakey process from the captured grayscale image, a subject-removed captured grayscale image is generated, as indicated by the white arrow. In the subject-removed captured grayscale image shown on the right side of Figure 24, the hatched subject area indicates that the subject area has been removed. The subject-removed captured grayscale image generated in this manner is used in signal processing 2.

[0126] Signal Processing 2 (Case where Moire is Present) FIG. 25 is a diagram showing an example of signal processing 2 when moire is present.

[0127] As shown on the left side of Fig. 25, the photographed background image is an image in which a background image such as a landscape is reflected in the background of the subject. Moire is reflected in the entire photographed background image.

[0128] In signal processing 2, a signal-processed captured image is generated as indicated by the white arrow, for example, by dividing the subject-removed grayscale captured image from the background captured image. The division is a process in which a moiré luminance distribution is calculated assuming that the average pixel value of the pixels in the entire grayscale captured image is 1, and the pixel value of each pixel in the background captured image is divided by the value of the moiré luminance distribution at the corresponding position. The signal-processed captured image may also be generated by a calculation other than division using the background captured image and the subject-removed grayscale captured image.

[0129] The subject-removed grayscale captured image generated by signal processing 1 is an image that mainly contains moiré components. By subtracting the moiré components from the background captured image, the background captured image from which the moiré components, which are interference components, have been removed is generated as a signal-processed captured image, as shown on the right side of Fig. 25 .

[0130] In this way, the above-described technique can also be applied to the removal of moire. For the three types of video used for moire removal, the frame frequency of each frame can be selected arbitrarily, and any frame configuration can be used.

[0131] <Others> Example of a frame configuration of a display image FIG. 26 is a diagram showing another example of a display image used for displaying on the LED display 11. In FIG.

[0132] The display image shown in Fig. 26 has a frame configuration in which frames of a background image, a black image, a gray image, and a monochrome image are arranged in this order. For each frame of the background image, one frame of a black image, one frame of a gray image, and one frame of a monochrome image are added. The four types of images shown in Fig. 26, which are made up of four types of images - a background image, a black image, a gray image, and a monochrome image - are display images that can both remove uneven reflections and glare and remove moiré.

[0133] The captured video synchronized with the display of the four types of video is composed of a background captured image, a black-image captured image, a gray-image captured image, and a monochrome captured image. Of the background captured image, black-image captured image, gray-image captured image, and monochrome captured image included in the captured video imported into the image processing device 51, signal processing 1 and signal processing 2 are performed using the background captured image, black-image captured image, and monochrome captured image to remove uneven reflections and glare. Furthermore, signal processing 1 and signal processing 2 are performed using the background captured image, gray-image captured image, and monochrome captured image to remove moire.

[0134] In this way, four types of images may be used to display on the LED display 11, and correction may be performed to remove both uneven reflection / glare and moire.

[0135] Example of System Configuration of Display System Although the display system is configured with the LED display 11, the video server 31, and the display controller 32, the configuration of the display system can be changed as appropriate.

[0136] For example, the display system may be configured by the display controller 32 alone, or may be configured by a signal processing chip within the display controller 32 .

[0137] The function of the display controller 32 (display control unit 211) that controls the display on the LED display 11 may be installed in the video server 31, and a display system may be configured by the video server 31 having this function. Alternatively, the function of the display controller 32 that controls the display on the LED display 11 may be installed in the LED display 11, and a display system may be configured by the LED display 11 having this function.

[0138] - Example of a program The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0139] The program to be installed is provided by being recorded on removable media 121 shown in FIG. 13, which may be an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 112 or storage unit 118.

[0140] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0141] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0142] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0143] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0144] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0145] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0146] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0147] Example of configuration combinations The present technology can also be configured as follows.

[0148] (1) A display system including a display control unit that causes a display used for displaying a background image to be displayed, the display having a frame configuration in which a background image, which is a frame image of a predetermined color, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color lower in saturation than the predetermined color, are arranged in a predetermined order. (2) The display system described in (1), wherein the display control unit displays the display image in which one frame of the first color image and one frame of the second color image are arranged between frames of the background image. (3) The display system described in (1) or (2), wherein the display control unit displays the display image including one frame of the background image in a set of the background image, the first color image, and the second color image. (4) The display system described in (1) or (2), wherein the display control unit displays the display image including multiple frames of the background image in a set of the background image, the first color image, and the second color image. (5) The display system described in any of (1) to (4), wherein the second color image is a black image. (6) The display system according to any one of (1) to (4), wherein the second color image is a white or gray image. (7) The display system according to any one of (1) to (6), wherein the display control unit displays each frame of the display video in synchronization with the timing of capturing an image of the subject by a camera. (8) The display system according to any one of (1) to (7), further comprising the display. (9) A program that causes a computer to execute a process of displaying, on a display used for displaying the background video, a display video in which a background image that is a frame image constituting a background video captured as the background of a subject, a first color image that is a frame image of a predetermined color, and a second color image that is a frame image of a color lower in saturation than the predetermined color, are arranged in a predetermined order.(10) An information processing system comprising: an acquisition unit that acquires captured video obtained by capturing a subject against a display that displays video having a frame configuration in which a background image that is a frame image constituting a background video, a first color image that is a frame image of a predetermined color, and a second color image that is a frame image of a color lower in saturation than the predetermined color, are arranged in a predetermined order; and an image processing unit that detects a region of the subject based on a first captured image captured in synchronization with the display timing of the first color image, removes the region of the subject from a second captured image captured in synchronization with the display timing of the second color image, and corrects the captured background image captured in synchronization with the display timing of the background image based on the second captured image from which the region of the subject has been removed. (11) The information processing system described in (10), wherein the second color image is a black image. (12) The information processing system described in (11), wherein the image processing unit removes a reflected image of the display included in the captured background image based on the second captured image from which the region of the subject has been removed. (13) The information processing system according to (10), wherein the second color image is a white or gray image. (14) The information processing system according to (13), wherein the image processing unit removes interference components contained in the background captured image based on the second captured image from which the subject region has been removed. (15) A program that causes a computer to execute processes of: acquiring captured video obtained by capturing a subject against a background display that displays video having a frame configuration in which a background image that is a frame image constituting a background video, a first color image that is a frame image of a predetermined color, and a second color image that is a frame image of a color lower in saturation than the predetermined color, are arranged in a predetermined order; detecting the subject region based on the first captured image captured in synchronization with the display timing of the first color image; removing the subject region from the second captured image captured in synchronization with the display timing of the second color image; and correcting the background captured image captured in synchronization with the display timing of the background image based on the second captured image from which the subject region has been removed.

[0149] REFERENCE SIGNS LIST 1 Photography system, 11 LED display, 12 Video camera, 21 Display unit, 31 Video server, 32 Display controller, 51 Image processing device, 201 Background image acquisition unit, 202 Three-type image generation unit, 203 Three-type image output unit, 211 Display control unit, 251 Captured image acquisition unit, 252 Image processing unit

Claims

1. A display system comprising a display control unit that displays a display image having a frame configuration in which a background image, which is a frame image constituting a background image photographed as the background of a subject, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order on a display used to display the background image.

2. The display system according to claim 1, wherein the display control unit displays the display image in which one frame of the first color image and one frame of the second color image are arranged between frames of the background image.

3. The display system according to claim 1, wherein the display control unit displays the display image including one frame of the background image in one set of the background image, the first color image, and the second color image.

4. The display system according to claim 1, wherein the display control unit displays the display image including a plurality of frames of the background image in one set of the background image, the first color image, and the second color image.

5. The display system of claim 1, wherein the second color image is a black image.

6. The display system according to claim 1, wherein the second color image is a white or gray image.

7. The display system according to claim 1, wherein the display control unit displays each frame of the display image in synchronization with the timing of the image capture of the subject by a camera.

8. The display system of claim 1 further comprising the display.

9. A program that causes a computer to execute a process to display, on a display used for displaying the background image, a display image in which a background image, which is a frame image constituting a background image photographed as the background of a subject, a first color image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order.

10. An information processing system comprising: an acquisition unit that acquires captured video obtained by capturing a subject against a display that displays a display video having a frame configuration in which a background image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color lower in saturation than the predetermined color, are arranged in a predetermined order; and an image processing unit that detects an area of ​​the subject based on a first captured image captured in synchronization with the display timing of the first color image, removes the area of ​​the subject from a second captured image captured in synchronization with the display timing of the second color image, and corrects the background captured image captured in synchronization with the display timing of the background image based on the second captured image from which the area of ​​the subject has been removed.

11. The information processing system according to claim 10, wherein the second color image is a black image.

12. The information processing system according to claim 11, wherein the image processing unit removes a reflected image of the display included in the background captured image based on the second captured image from which the subject area has been removed.

13. The information processing system according to claim 10, wherein the second color image is a white or gray image.

14. The information processing system according to claim 13, wherein the image processing unit removes interference components contained in the background captured image based on the second captured image from which the subject area has been removed.

15. A program that causes a computer to execute the following processes: acquiring a captured image obtained by capturing an object against a display that displays an image having a frame configuration in which a background image, which is a frame image of a predetermined color, and a second color image, which is a frame image of a color less saturated than the predetermined color, are arranged in a predetermined order; detecting an area of ​​the object based on a first captured image captured in synchronization with the display timing of the first color image; removing the area of ​​the object from a second captured image captured in synchronization with the display timing of the second color image; and correcting the background captured image captured in synchronization with the display timing of the background image based on the second captured image from which the area of ​​the object has been removed.

Images