CONTROL DEVICE, PHOTOGRAPHY SYSTEM, AND 3D MODEL DATA GENERATION METHOD

Abstract

The present technology relates to a control device, an image capture system, and a 3D model data generating method for making it possible to appropriately extract a subject from a moving image. The control device according to the present technology comprises: a control unit which performs control to synchronize an image capture timing of the image capture device performing image capture and a display timing of a display device, positioned in an image capture range of the image capture device, displaying a first moving image; and an image processing unit which performs a foreground / background distinguishing process on the basis of a second moving image captured by the image capture device at the first timing and a third moving image captured by the image capture device at a second timing different from the first timing. The second moving image and the third moving image each include at least a part of the first moving image of which the display timing is synchronized between the second moving image and the third moving image. The present technology is applicable to, for example, an image capture system for capturing a moving image for use in 3D model generation.

Description

[Technical field]

[0001] The present technology relates to a control device, an imaging system, and a 3D model data generation method, and in particular to a control device, an imaging system, and a 3D model data generation method that are capable of appropriately extracting a subject from a moving image. [Background technology]

[0002] There is a technology that generates a 3D model of a subject from video images captured from multiple viewpoints, and generates a virtual viewpoint image of the 3D model according to an arbitrary viewpoint position to provide an image from any viewpoint. This type of technology is also called volumetric capture technology.

[0003] For example, Patent Document 1 describes a technique for generating a 3D model using a method such as Visual Hull, which carves out a three-dimensional shape of a subject based on a plurality of images captured from different angles. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 150933 Summary of the Invention [Problem to be solved by the invention]

[0005] In generating a 3D model, it is necessary to separate the subject and background in each image captured from multiple viewpoints. For example, the subject and background in each image can be separated by taking the difference between a background image obtained by capturing only the background in advance and an image captured when a subject such as a person actually exists.

[0006] By the way, when filming to generate a 3D model, there may be situations where the subject of the shoot will want to check the content of the performance by displaying moving images on a display device, or refer to lines or lyrics on a teleprompter.

[0007] However, when a person is present during actual shooting, if there is something in the shooting space that was not present during background shooting, that thing itself will be treated as a subject. For example, when a moving image is displayed on a display device placed in the shooting space, the display of the moving image is also separated as a subject together with the person and used to generate a 3D model. Therefore, during actual shooting, it is difficult for the subject to check the content of the performance by referring to the display on the display device, or to check the lines or lyrics by referring to the display on the teleprompter.

[0008] The present technology has been made in consideration of such circumstances, and makes it possible to appropriately extract a subject from a moving image. [Means for solving the problem]

[0009] A control device according to one aspect of the present technology includes a control unit that controls synchronization between a shooting timing at which an imaging device captures an image and a display timing at which a display device located in the shooting range of the imaging device displays a first moving image, and an image processing unit that performs foreground-background differential processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing that is different from the first timing, wherein the second moving image and the third moving image both include at least a portion of the first moving image whose display timing is synchronized with the second moving image and the third moving image.

[0010] A photography system according to one aspect of the present technology includes a plurality of imaging devices, at least one display device, a control unit that controls synchronization between a shooting timing at which the plurality of imaging devices capture images and a display timing at which the display device located in the shooting range of at least one of the plurality of imaging devices displays a first moving image, and an image processing unit that performs foreground-background differential processing based on a second moving image captured by the imaging devices at a first timing and a third moving image captured by the imaging devices at a second timing that is different from the first timing, and the second moving image and the third moving image both include at least a portion of the first moving image.

[0011] A 3D model data generation method according to one aspect of the present technology is a 3D model data generation method that captures images using a plurality of imaging devices by synchronizing a capture timing at which the imaging devices capture images with a display timing at which a display device located in a capture range of at least one of the plurality of imaging devices displays a first moving image, generates difference data by performing foreground-background difference processing based on a second moving image captured by the imaging devices at the first timing and a third moving image captured by the imaging devices at a second timing that is different from the first timing, and generates 3D model data based on the difference data.

[0012] In a control device according to one aspect of the present technology, a control is performed to synchronize a capture timing at which an imaging device captures an image with a display timing at which a display device located in a capture range of the imaging device displays a first moving image, and a foreground-background subtraction process is performed based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing. The second moving image and the third moving image both include at least a portion of the first moving image whose display timing is synchronized with that of the second moving image and the third moving image.

[0013] In the imaging system according to one aspect of the present technology, a control is performed to synchronize a capture timing at which the multiple imaging devices capture images with a display timing at which the display device located in the imaging range of at least one of the multiple imaging devices displays a first moving image, and a foreground-background subtraction process is performed based on a second moving image captured by the imaging device at the first timing and a third moving image captured by the imaging device at a second timing different from the first timing. The second moving image and the third moving image both include at least a part of the first moving image.

[0014] In a method for generating 3D model data according to one aspect of the present technology, multiple imaging devices capture images by synchronizing a capture timing at which the multiple imaging devices capture images with a display timing at which a display device located in the capture range of at least one of the multiple imaging devices displays a first moving image, and foreground-background subtraction processing is performed based on a second moving image captured by the imaging devices at the first timing and a third moving image captured by the imaging devices at a second timing different from the first timing, to generate difference data, and 3D model data is generated based on the difference data. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an imaging system to which the present technology is applied. [Diagram 2] FIG. 1 is a diagram showing an overview of 3D model generation executed by the control device. [Diagram 3] FIG. 11 is a plan view showing an example of a situation when the photography system with N=8 photographs the background. [Figure 4] FIG. 11 is a plan view showing an example of a situation when an imaging system with N=8 performs actual imaging. [Diagram 5] 11A and 11B are diagrams illustrating an example of control of image capture timing and display timing based on a synchronization signal. [Figure 6] FIG. 13 is a diagram illustrating an example of foreground / background subtraction processing. [Figure 7]FIG. 1 is a block diagram showing an example of the configuration of an imaging system. [Figure 8] 13 is a flowchart illustrating a synchronized moving image generating process of the control device. [Figure 9] 11 is a flowchart illustrating a moving image shooting process. [Figure 10] 11 is a flowchart illustrating a process of generating 3D model data of a subject by a control device. [Figure 11] 13 is a flowchart illustrating details of a foreground / background subtraction process. [Figure 12] FIG. 11 is a block diagram showing an example of the configuration of an imaging system when performing monitoring display. [Figure 13] 13 is a flowchart illustrating a synchronized moving image generating process of the control device when performing monitoring display. [Figure 14] FIG. 13 is a diagram illustrating another example of the configuration of the imaging system. [Figure 15] FIG. 15 is a plan view showing a state during actual shooting in the shooting system of FIG. 14 where N=8. [Figure 16] FIG. 13 is a diagram illustrating yet another example configuration of the imaging system. [Figure 17] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of the imaging system 2. Imaging system configuration 3. Operation of the imaging system 4. Processing when performing monitoring display 5. Variations 6. Computer configuration examples

[0017] <1. Overview of the imaging system> FIG. 1 is a diagram showing an example of the configuration of an imaging system 1 to which the present technology is applied.

[0018] 1 includes a control device 11, N (N>1) cameras 12-1 to 12-N, and a monitor 13. The cameras 12-1 to 12-N and the monitor 13 are arranged in the same space, such as a photography studio.

[0019] The control device 11 is configured, for example, by a computer or a server device, and controls the timing at which the cameras 12-1 to 12-N capture images and the display timing at which the monitor 13 displays moving images.

[0020] Specifically, the control device 11 generates a synchronization signal and supplies it to each of the cameras 12-1 to 12-N and the monitor 13. Based on the synchronization signal supplied from the control device 11, the capture timing of the cameras 12-1 to 12-N to capture images and the display timing of the monitor 13 to display moving images are controlled. The control device 11 also plays back the moving images displayed on the monitor 13.

[0021] Furthermore, the control device 11 generates a 3D model of an object from a plurality of captured moving images supplied from each of the cameras 12-1 to 12-N, with, for example, a person appearing as a subject being taken as the object.

[0022] The data of the 3D model of the object is composed of, for example, video data of the object generated from the captured video captured by the cameras 12-1 to 12-N, and 3D shape data representing the 3D shape of the object. The 3D shape data is represented, for example, by a depth image corresponding to the captured video captured by the multiple cameras 12-1 to 12-N, a point cloud representing the 3D position of the object as a set of points, or a polygon mesh representing vertices and connections between the vertices.

[0023] The cameras 12-1 to 12-N are imaging devices that capture a subject to be extracted as an object and its background under the control of the control device 11. The cameras 12-1 to 12-N supply captured moving images obtained by capturing images to the control device 11. The cameras 12-1 to 12-N are arranged, for example, to surround the same subject so as to capture the same subject. The captured moving images supplied from the cameras 12-1 to 12-N are multi-viewpoint moving images made up of a plurality of captured moving images from different viewpoints from which the subject is captured.

[0024] In the following description, when there is no need to distinguish between the cameras 12-1 to 12-N, they will simply be referred to as the camera 12. The same applies to other configurations in which a plurality of cameras are provided.

[0025] Monitor 13 is a display device that displays a reference moving image as a first moving image under the control of control device 11. The reference moving image is a moving image that a person who is a subject uses as a reference during actual shooting. For example, during actual shooting, a moving image that serves as a model for the movements of the person who is a subject is displayed on monitor 13 as a reference moving image.

[0026] The monitor 13 can also perform monitoring display, which displays monitoring images as reference moving images. The monitoring images are moving images captured in real time by any of the cameras 12-1 to 12-N. In the monitoring display, the subject can take a predetermined action while checking his or her own movements.

[0027] Furthermore, a moving image showing lines or lyrics can be displayed as a reference moving image on the monitor 13. By displaying a moving image showing lines or lyrics, the subject can act while checking the lines or sing while checking the lyrics.

[0028] The monitor 13 may be configured by a device such as a smartphone or a tablet terminal (mobile terminal) equipped with a display.

[0029] The communication between the control device 11 and the cameras 12-1 to 12-N, and the communication between the control device 11 and the monitor 13 may be performed directly via a cable or the like, or may be performed via a predetermined network such as a LAN (Local Area Network) or the Internet. Furthermore, the communication may be wired communication or wireless communication.

[0030] Next, with reference to FIG. 2, an overview of the 3D model generation executed by the control device 11 will be described.

[0031] Fig. 2 shows an example in which a person performing a predetermined action is photographed as subject Ob1 using three cameras 12. As shown on the left side of Fig. 2, three cameras 12-1 to 12-3 arranged to surround subject Ob1 photograph subject Ob1.

[0032] Using the captured video images obtained from multiple cameras 12 arranged at different positions, 3D modeling is performed by the control device 11, and a 3D model MO1 of the subject Ob1 is generated as shown in the center of Fig. 2. The 3D model MO1 is generated, for example, by a method such as Visual Hull, which cuts out a three-dimensional shape using the captured video images obtained by photographing the subject Ob1 from different directions.

[0033] The data of the 3D model of the subject (3D model data) generated as described above is transmitted to the playback device and played back. That is, the playback device renders the 3D model based on the 3D model data, and a 3D shape image is displayed on the viewing device. In Fig. 2, a display D1 and a head mounted display (HMD) D2 are shown as viewing devices used by the viewer.

[0034] In generating a 3D model, it is necessary to separate the subject from the background in each captured video and extract data of only the subject. Keying using a green screen background is known as a method for extracting data of only the subject. In addition, the subject can be separated from the background by using foreground-background subtraction processing to find the difference between a background video, which is a video in which only the background is captured, and a live video, which is a video in which the subject and background are captured.

[0035] The photography system 1 in Fig. 1 performs background photography, which is photography of a background moving image, and actual photography, which is photography of an actual moving image, and separates the subject from the background by calculating the difference between the background moving image and the actual moving image through foreground-background difference processing. Here, actual photography is photography in a situation where a subject is present, whereas background photography is photography in a situation where a subject is not present. The actual moving image is a second moving image photographed by the camera 12 at a predetermined first timing (timing of actual photography), and the background moving image is a third moving image photographed by the camera 12 at a second timing (timing of background photography) that is different from the first timing.

[0036] FIG. 3 is a plan view showing an example of a situation in which the photographing system 1, where N=8, photographs the background.

[0037] 3, eight cameras 12-1 to 12-8 are arranged in a ring shape at approximately equal intervals toward the center of a shooting space such as a room. The cameras 12-1 and 12-5, the cameras 12-2 and 12-6, the cameras 12-3 and 12-7, and the cameras 12-4 and 12-8 are arranged facing each other.

[0038] Monitor 13 is placed between cameras 12-5 and 12-6 with its front facing toward the center. Here, monitor 13 is located within the shooting range of camera 12-2. As shown in speech bubble #1, monitor 13 displays a video of a person dancing as a reference video.

[0039] In this way, background shooting is performed in a state where no subject is present. In the background moving images shot by each of the cameras 12-1 to 12-8, other cameras 12, the monitor 13, and the walls, ceiling, and floor of the shooting space that are present within the angle of view are captured as background.

[0040] For example, as shown in speech bubble #2, in the background video captured by camera 12-2, camera 12-6 appears near the center, and monitor 13 appears to the left of camera 12-6. Since the reference video is displayed on monitor 13 during background capture, the reference video showing people dancing is also displayed on monitor 13 that appears in the background video captured by camera 12-2.

[0041] FIG. 4 is a plan view showing an example of a situation when the imaging system 1 performs actual imaging with N=8.

[0042] As shown in FIG. 4, the arrangement of the cameras 12-1 to 12-8 and the monitor 13 during actual shooting is the same as that during background shooting described with reference to FIG.

[0043] The actual shooting is performed in a situation where a person as the subject Ob11 is present in the center of the shooting space. The actual moving images shot by each of the cameras 12-1 to 12-8 will include the subject Ob11 and the background present within the angle of view. During the actual shooting, as shown in the speech bubble #11, the same moving image as that displayed during the background shooting is displayed on the monitor 13 as the reference moving image.

[0044] As shown in speech bubble #12, in the actual video shot by camera 12-2, subject Ob11 appears near the center, and monitor 13 appears to the left of subject Ob11. Since the reference video is displayed on monitor 13 during actual shooting, the reference video showing people dancing is also displayed on monitor 13 that appears in the actual video shot by camera 12-2.

[0045] In this way, the actual shooting is carried out in the same manner as the background shooting, except that the subject is placed in the center of the shooting space.

[0046] When separating the subject from the background using foreground-background subtraction, if something that was not present when the background was photographed is present in the shooting space during the actual shoot, that thing itself will be treated as a subject.

[0047] For example, if the reference video image of monitor 13 captured in the actual video image captured by camera 12-2 during actual shooting and background shooting differs from the reference video image displayed on monitor 13 during background shooting, the reference video image of monitor 13 captured in the actual video image will also be extracted as a subject.

[0048] Therefore, the control device 11 of the photographing system 1 generates a synchronization signal and supplies it to the cameras 12-1 to 12-8 and the monitor 13, thereby controlling so that the photographing timing of the cameras 12-1 to 12-8 and the display timing of the monitor 13 are synchronized.

[0049] FIG. 5 is a diagram showing an example of control of image capture timing and display timing based on a synchronization signal.

[0050] During background shooting and actual shooting, for example, a synchronization signal as shown in the upper part of Fig. 5 is supplied to each of the camera 12 and the monitor 13. The synchronization signal shown in Fig. 5 is a pulse signal that alternates between 0 (Low) and 1 (High) at a predetermined cycle, and in the example of Fig. 5, it rises at times t1, t3, t5, t7,... and falls at times t2, t4, t6, t8,...

[0051] The camera 12 performs exposure (shooting) to obtain frame images constituting a moving image (main moving image or background moving image) at each of times t1, t3, t5, and t7, which are rising edges of a synchronization signal.

[0052] The monitor 13 sequentially displays one or more frame images constituting the reference moving image at each of the rising edges of the synchronization signal, t1, t3, t5, and t7. As described above, the reference moving image is displayed during each of the main shooting and background shooting.

[0053] The control device 11 decodes the frame images prior to the update timing (display timing) of the frame images displayed on the monitor 13. For example, as shown in Fig. 5, the frame image displayed at time t1 is decoded before time t1. Similarly, the frame images displayed at times t3, t5, and t7 are decoded before the respective times.

[0054] During both actual shooting and background shooting, the control device 11 generates a synchronization signal and supplies it to the cameras 12-1 to 12-8 and the monitor 13, so that at least a portion of the same frame image of the reference moving image is captured in the frame images of the same frame numbers of the actual moving image and the background moving image.

[0055] Since the same frame images of the reference video are reflected in frame images of the same frame numbers in the actual video and the background video, when separating the subject from the background by foreground-background subtraction processing, the area of ​​the reference video displayed on the monitor 13 in the actual video can be treated as the background. This makes it possible to prevent the reference video of the monitor 13 reflected in the actual video from being extracted as the subject. As a result, in the shooting system 1, it becomes possible to shoot while providing the reference video to the person who is the subject in shooting for generating a 3D model.

[0056] In the above example, the camera 12 captures the image and the monitor 13 displays the reference video at the rising edge of the synchronization signal, but the camera 12 may capture the image and the monitor 13 may display the reference video at the falling edge of the synchronization signal.

[0057] FIG. 6 is a diagram illustrating an example of foreground / background subtraction processing.

[0058] The actual moving image shown on the left side of Fig. 6 is the actual moving image captured by the camera 12-2 described with reference to Fig. 4. Moreover, the background moving image shown in the center of Fig. 6 is the background moving image captured by the camera 12-2 described with reference to Fig. 3.

[0059] The control device 11 obtains the difference between the main moving image and the background moving image, and extracts data of the area in which the subject Ob11 appears as difference data, as shown on the right side of FIG.

[0060] Specifically, the control device 11 compares the pixel values ​​of pixels constituting the actual moving image with those of pixels in the background moving image at the same pixel positions as the pixels constituting the actual moving image, and extracts pixels in the actual moving image whose pixel value difference is equal to or exceeds a predetermined threshold value as foreground pixels. This pixel value comparison is performed for all pixels constituting the actual moving image, and the foreground pixels, i.e., the pixels constituting the subject, are generated as difference data.

[0061] The pixel values ​​may be compared by comparing RGB values, or may be compared by comparing HSV (Hue, Saturation, Value) values ​​obtained by converting the RGB values. Comparison using HSV values ​​enables control not to extract a change in pixel value in which only brightness changes without a change in color (hue), such as the presence or absence of a shadow, as a change. In addition, difference data may be generated by comparing pixel values ​​not only of one pixel to be compared, but also of pixels surrounding the pixel to be compared. The threshold value used to determine the foreground pixel may be set appropriately according to the shooting environment.

[0062] The foreground-background subtraction process described above is performed on the actual video images and background video images captured by the multiple cameras 12, and difference data for the video images is generated for each camera 12.

[0063] As described above, the shooting system 1 makes it possible to appropriately extract a subject from an actual moving image in which the monitor 13 displaying the reference moving image is captured. Even if the image displayed on the monitor 13 changes during shooting, it is possible to perform foreground-background subtraction processing without being affected by the change.

[0064] <2. Configuration of the imaging system> FIG. 7 is a block diagram showing an example of the configuration of the imaging system 1. As shown in FIG.

[0065] The photographing system 1 includes a control device 11, N cameras 12-n (n=1 to N), and a monitor 13. The camera 12-n is provided with a photographing unit 51. The monitor 13 is provided with a display unit 52.

[0066] The control device 11 is made up of an operation unit 31 , a control unit 32 , a playback unit 33 , a synchronization signal generating unit 34 , a memory control unit 35 , a memory unit 36 ​​, and an image processing unit 37 .

[0067] The operation unit 31 is composed of a touch panel monitor, a keyboard, a mouse, a controller, a remote control device, etc. The operation unit 31 detects operations by the photographer or a person who is to be a subject, and supplies information indicating the contents of the detected operations to the control unit 32.

[0068] For example, the operation unit 31 detects an operation for starting background shooting, an operation for starting actual shooting, an operation for generating 3D model data, etc. The operation unit 31 may detect an operation for designating a reference video image to be displayed on the monitor 13 during background shooting and actual shooting.

[0069] Based on information supplied from the operation unit 31, the control unit 32 controls each component of the control device 11 and the camera 12-n.

[0070] For example, when an operation to start actual shooting or background shooting is detected by the operation unit 31, the control unit 32 controls the shooting unit 51 of the camera 12-n to perform various settings such as the frame rate and resolution of the video. The control unit 32 also controls the playback unit 33 to display a predetermined reference video on the monitor 13. The control unit 32 also controls the synchronization signal generation unit 34 to generate a synchronization signal.

[0071] On the other hand, when an operation for generating 3D model data is detected by the operation unit 31, the control unit 32 controls the image processing unit 37 to perform a 3D model data generation process.

[0072] The playback unit 33 acquires and plays back a reference video designated by the control unit 32 from among the reference videos stored in the storage unit 36. The playback unit 33 decodes the reference video prior to the display timing of the monitor 13, and supplies the reference video obtained by decoding to the display unit 52. The reference video is decoded by a method corresponding to the encoding method used when storing the reference video in the storage unit 36. It should be noted that the frame rate of the reference video is the same as the frame rate used when the shooting unit 51 performs actual shooting or background shooting.

[0073] The synchronization signal generating unit 34 generates the synchronization signal described with reference to FIG. 5 under the control of the control unit 32, and supplies it to the photographing unit 51 and the display unit 52, respectively.

[0074] The storage control unit 35 encodes the moving images supplied from the imaging unit 51 using a predetermined encoding method such as the MPEG2 method or the AVC method, and stores the encoded images in the storage unit 36.

[0075] The storage unit 36 ​​is configured by an auxiliary storage device including an internal or external storage such as a semiconductor memory, etc. The storage unit 36 ​​has a playback data storage unit 41 and a shooting data storage unit 42.

[0076] The playback data storage unit 41 stores reference moving images to be played back by the playback unit 33 .

[0077] The shooting data storage unit 42 stores the background moving image and the actual moving image supplied from the storage control unit 35. The shooting data storage unit 42 also stores information about the camera 12-n used to shoot the moving images, together with the background moving image and the actual moving image. The information about the camera 12-n includes, for example, the internal parameters and external parameters of the camera 12-n.

[0078] The reproduction data storage section 41 and the shooting data storage section 42 may be provided in the control device 11 as separate storage sections.

[0079] The image processing unit 37 generates 3D model data of the subject under the control of the control unit 32. The image processing unit 37 has a foreground / background difference processing unit 43 and a 3D model data generation unit .

[0080] The foreground / background difference processing unit 43 acquires the actual moving image and the background moving image from the storage unit 36 ​​for each camera 12. The foreground / background difference processing unit 43 performs foreground / background difference processing using the actual moving image and the background moving image to generate difference data. The difference data generated by the foreground / background difference processing unit 43 is supplied to the 3D model data generation unit 44.

[0081] The 3D model data generation unit 44 acquires information about each camera 12 from the storage unit 36. The 3D model data generation unit 44 generates 3D model data of the subject using a method such as Visual Hull based on the difference data for each camera 12 supplied from the foreground / background difference processing unit 43 and the information about each camera 12. The 3D model data of the subject generated by the 3D model data generation unit 44 is stored in the storage unit 36 ​​or output to an external device.

[0082] The image processing unit 37 may be provided as an image processing device separate from the control device 11, rather than as a part of the control device 11.

[0083] The imaging unit 51 of the camera 12-n captures the actual moving image or the background moving image at a timing based on the synchronization signal supplied from the synchronization signal generating unit 34. The moving image acquired by capturing the actual moving image or the background moving image is supplied to the storage control unit 35 of the control device 11.

[0084] The display unit 52 of the monitor 13 displays the reference moving image supplied from the playback unit 33 at a display timing based on the synchronization signal supplied from the synchronization signal generation unit .

[0085] <3. Operation of the imaging system> Next, the operation of the photographing system 1 will be described with reference to the flowcharts of FIGS.

[0086] First, the synchronized moving image generating process of the control device 11 will be described with reference to the flowchart of FIG.

[0087] 8 is a series of processes for shooting a background video image and a main video image. This process is started, for example, when an operation for starting background shooting is detected by the operation unit 31.

[0088] In step S1, the control device 11 performs a background moving image capturing process. The background moving image capturing process is a process in which the moving image capturing process in FIG. 9 described later is performed in a state where no subject is present in the capturing space. By the background moving image capturing process, a background moving image is generated and stored in the storage unit 36.

[0089] Next, when an operation to start actual shooting is detected by the operation unit 31, the control device 11 performs actual video shooting processing in step S2. The actual video shooting processing is processing in which the video shooting processing in Fig. 9 described later is performed in a state in which a subject is present in the shooting space. The actual video shooting processing generates an actual video and stores it in the storage unit 36. When the background video and the actual video are stored in the storage unit 36, the synchronized video generation processing ends.

[0090] Next, the moving image shooting process executed as the process in step S1 or S2 in FIG. 8 will be described with reference to the flowchart in FIG.

[0091] In step S11, the control unit 32 prepares the camera 12 and the monitor 13. Specifically, the control unit 32 controls the shooting unit 51 and performs various settings such as the frame rate and resolution of the video. The control unit 32 also controls the playback unit 33 to display the reference video on the monitor 13. The playback unit 33 acquires the reference video specified by the control unit 32, decodes the reference video, and appropriately supplies frame images of the reference video to the display unit 52.

[0092] In step S12, the control unit 32 causes the synchronization signal generation unit 34 to start generating a synchronization signal. The synchronization signal generation unit 34 generates a synchronization signal under the control of the control unit 32, and supplies the synchronization signal to the shooting unit 51 and the display unit 52 of each camera 12.

[0093] In step S13, the shooting unit 51 of each camera 12 shoots a background moving image or an actual moving image based on the synchronization signal supplied from the synchronization signal generating unit 34. At the same time, the display unit 52 displays the reference moving image based on the synchronization signal supplied from the synchronization signal generating unit 34.

[0094] In step S14, the control unit 32 causes the synchronization signal generating unit 34 to stop generating the synchronization signal. For example, the process of step S14 is performed when an operation to end shooting is detected by the operation unit 31. The process of step S14 may be performed when a predetermined time has elapsed since the generation of the synchronization signal was started.

[0095] In step S15, the control unit 32 receives a notification from the imaging unit 51 indicating that imaging has ended.

[0096] In step S16, the storage control unit 35 stores the moving images captured by each camera 12 in the storage unit 36. That is, the storage control unit 35 encodes the background moving images or the actual moving images captured by the imaging unit 51 in the process of step S13 and supplied thereto using a predetermined encoding method such as the MPEG2 method or the AVC method, and stores the encoded images in the storage unit 36.

[0097] This ends the video shooting process executed as the process of step S1 or step S2 in Fig. 8. The difference between the background video shooting process of step S1 in Fig. 8 and the actual video shooting process of step S2 is whether or not they are performed in a state where a subject is present in the shooting space.

[0098] Next, a process for generating 3D model data of a subject by the control device 11 will be described with reference to the flowchart of Fig. 10. This process is started, for example, when an operation for generating 3D model data of a subject is detected by the operation unit 31.

[0099] First, in step S31, foreground / background subtraction processing unit 43 performs foreground / background subtraction processing for each actual moving image captured by each camera 12. By the foreground / background subtraction processing, difference data of the moving image for each camera 12 is generated. Details of the foreground / background subtraction processing will be described later with reference to the flowchart of FIG.

[0100] In step S32, the 3D model data generating unit 44 generates 3D model data of the subject based on the difference data of the moving images for each camera 12.

[0101] The foreground / background subtraction process performed in step S31 of FIG. 10 will be described in detail with reference to the flowchart of FIG.

[0102] In step S41, foreground / background differential processing unit 43 reads a predetermined frame image from among the multiple frame images constituting the actual moving image from storage unit 36. For example, in the process of step S41 which is repeatedly executed as described below, foreground / background differential processing unit 43 reads the multiple frame images constituting the actual moving image one frame at a time in order, starting from the first frame image.

[0103] In step S42, the foreground background difference processing unit 43 reads from the memory unit 36 ​​a frame image of the background moving image that has the same frame number as the frame image of the actual moving image read in step S41, among the multiple frame images that make up the background moving image.

[0104] In step S43, the foreground / background difference processing unit 43 sets a predetermined pixel among the pixels constituting the frame image of the actual moving image read from the storage unit 36 ​​as a processing target pixel, and calculates the pixel value difference between the processing target pixel and a pixel corresponding to the processing target pixel among the pixels constituting the frame image of the background moving image. Then, the foreground / background difference processing unit 43 determines whether the calculated pixel value difference is equal to or greater than a predetermined threshold value.

[0105] If it is determined in step S43 that the difference in pixel value is equal to or greater than the predetermined threshold, the process proceeds to step S44, where foreground / background differential processing unit 43 sets the processing target pixel to a pixel in the foreground.

[0106] On the other hand, if it is determined in step S43 that the difference in pixel value is less than the predetermined threshold, the process proceeds to step S45, where the foreground / background differential processing unit 43 sets the processing target pixel to a background pixel.

[0107] In step S46, the foreground / background difference processing unit 43 determines whether or not there is a next pixel to be set as the processing target pixel among the pixels of the frame images of the actual moving image read in step S41. In step S46, if all the pixels of the frame images of the actual moving image read in step S41 have been set as the processing target pixel, it is determined that there is no next pixel to be set as the processing target pixel, and if all the pixels have not yet been set as the processing target pixel, it is determined that there is a next pixel to be set as the processing target pixel.

[0108] If it is determined in step S46 that there is a next pixel, the process returns to step S43, a pixel that has not yet been set as the processing target pixel is set as the processing target pixel, and the subsequent processes are carried out.

[0109] On the other hand, if it is determined in step S46 that there is no next pixel, the process proceeds to step S47, where foreground / background subtraction processing unit 43 generates a frame image of only the foreground. That is, in step S47, a frame image is generated that includes only the pixels set as foreground pixels among the frame images of the actual moving image read in step S41. A fixed pixel value such as 0 is stored in the pixels set as background pixels.

[0110] Next, in step S48, foreground / background differential processing unit 43 determines whether there is a next frame image, that is, whether there is a frame image in the actual moving image for which generation of a foreground-only frame image has not yet been performed.

[0111] If it is determined in step S48 that there is a next frame image, the process returns to step S41, and a frame image of the actual moving image for which a foreground-only frame image has not yet been generated is read as the next frame image, and subsequent processing is carried out.

[0112] On the other hand, if it is determined in step S48 that there is no next frame image, the process proceeds to step S49, where foreground / background differential processing unit 43 generates difference data of the moving image. That is, foreground / background differential processing unit 43 generates difference data of the moving image by arranging the foreground-only frame images generated in the repeatedly executed process of step S47 in the same order as the frame images of the actual moving image.

[0113] This completes the foreground background difference process in Fig. 11. The foreground background difference process in Fig. 11 is performed on each of the actual moving images captured by the multiple cameras 12. The actual moving images corresponding to each camera 12 may be selected one by one in turn and the foreground background difference process may be performed, or two or more or all of the actual moving images may be selected simultaneously and the foreground background difference process may be performed. After the difference data of the moving images is generated for all the actual moving images corresponding to all the cameras 12, the process returns to step S31 in Fig. 10 and the subsequent processes are performed.

[0114] According to the above-described shooting system 1, it becomes possible to shoot a person who is to be the subject while checking the reference moving image displayed on the monitor 13 during the actual shooting, and at that time, it is possible to prevent the moving image displayed on the monitor 13 from being extracted as the subject.

[0115] <4. Processing when performing monitoring display> Next, a case where the imaging system 1 captures a moving image while performing monitoring display will be described.

[0116] When the photography system 1 performs monitoring display, as described above, the video being photographed by the camera 12 is displayed in real time as a reference video on the monitor 13. That is, during actual photography, the actual video being photographed is displayed as it is on the monitor 13 as a reference video.

[0117] In this case, since the actual shooting is performed before the background shooting and the background shooting is performed after the actual shooting, the actual shooting image is displayed on the monitor 13 as a reference moving image during the background shooting. During the background shooting, the actual moving image displayed on the monitor 13 during the actual shooting and stored in the control device 11 is displayed as a reference moving image.

[0118] <Configuration of the Imaging System 1> FIG. 12 is a block diagram showing an example of the configuration of the imaging system 1 when performing monitoring display.

[0119] 12, the same components as those in the imaging system 1 in FIG. 7 are denoted by the same reference numerals, and duplicated explanations will be omitted as appropriate.

[0120] The configuration of the imaging system 1 shown in FIG. 12 differs from the configuration of the imaging system 1 described with reference to FIG.

[0121] Camera 12-m (m is any one of 1 to N) is a camera that shoots an actual moving image as a reference moving image to be displayed on monitor 13. The same synchronization signal as that for camera 12-n (n is 1 to N excluding m) is supplied from synchronization signal generator 34 to camera 12-m.

[0122] Like camera 12-n, camera 12-m is provided with an imaging unit 51-m. The imaging unit 51-m captures a main video or a background video at a timing based on a synchronization signal supplied from a synchronization signal generating unit 34. During main video capture, the main video captured by the imaging unit 51-m is supplied to the storage control unit 35 and the display unit 52. During background video capture, the background video captured by the imaging unit 51-m is supplied to the storage control unit 35 without being supplied to the display unit 52.

[0123] During actual shooting, the display unit 52 displays the actual moving image supplied from the shooting unit 51-m as a reference moving image, with a display timing based on the synchronization signal supplied from the synchronization signal generation unit 34. On the other hand, during background shooting, the display unit 52 displays the reference moving image supplied from the playback unit 33, with a display timing based on the synchronization signal supplied from the synchronization signal generation unit 34. The reference moving image supplied from the playback unit 33 is the same as the actual moving image supplied from the shooting unit 51-m during actual shooting.

[0124] The control unit 32 controls the camera 12-m in the same manner as the camera 12-n, based on information supplied from the operation unit 31. When the operation unit 31 detects an operation for starting actual shooting or background shooting, the control unit 32 controls the shooting units 51-m and 51-n to perform various settings such as the frame rate and resolution of the moving image.

[0125] Furthermore, during actual shooting, the control unit 32 does not cause the playback unit 33 to play back a predetermined video, because the actual video shot by the shooting unit 51-m is displayed on the monitor 13 as a reference video. In other words, during actual shooting, the control unit 32 controls the playback unit 33 not to play back a predetermined video. On the other hand, during background shooting, the control unit 32 controls the playback unit 33 to display again on the monitor 13 the actual video displayed on the monitor 13 during actual shooting as a reference video.

[0126] During actual shooting, the playback unit 33 does not play back a predetermined video in accordance with the control of the control unit 32. On the other hand, during background shooting, the playback unit 33 acquires and plays back a real video (a real video shot by the shooting unit 51-m) designated by the control unit 32 from the real videos stored in the storage unit 36. As in FIG. 7, the playback unit 33 decodes the reference video prior to the display timing of the monitor 13 and supplies the real video as a reference video to the display unit 52.

[0127] The storage control unit 35 encodes the moving images supplied from the imaging units 51-m and 51-n using a predetermined encoding method, and stores the encoded images in the storage unit 36.

[0128] In the reproduction data storage unit 41, the actual moving image captured by the imaging unit 51-m is stored as a reference moving image.

[0129] <Operation of the control device 11> Next, the synchronized moving image generating process of the control device 11 in the case of performing monitoring display will be described with reference to the flowchart of FIG.

[0130] The synchronized moving image generating process in FIG. 13 is started, for example, when the operation unit 31 detects an operation for starting actual shooting.

[0131] First, in step S101, the control device 11 performs a real moving image shooting process. In the process of step S101, the real moving image shot by the camera 12-m is displayed on the monitor 13, and the moving image shooting process of Fig. 9 is performed in a state where a subject exists in the shooting space. By the real moving image shooting process, the real moving image shot by each camera 12 is stored in the storage unit 36.

[0132] Next, when an operation for starting background shooting is detected by the operation unit 31, the control device 11 performs background moving image shooting processing in step S102. In the processing of step S102, the actual moving image displayed on the monitor 13 in the processing of step S101 is displayed on the monitor 13 as a reference moving image, and the moving image shooting processing of Fig. 9 is performed in a state where no subject is present in the shooting space. By the background moving image shooting processing, the background moving images shot by each camera 12 are stored in the storage unit 36.

[0133] 3D model data of a subject is generated using the actual video and the background video generated by the synchronized video generation process of Fig. 13. Note that the process of generating the 3D model data of a subject is similar to the process of generating the 3D model data of a subject shown in Fig. 10, and therefore a description thereof will be omitted.

[0134] According to the above-described shooting system 1, it is possible to generate the moving images necessary for generating 3D model data while the subject can check his / her own movements during the actual shooting, and at that time, it is possible to prevent the moving images displayed on the monitor 13 from being extracted as the subject.

[0135] <5. Modifications> <Example of timecode display> During each of the main shooting and background shooting, the control unit 32 can perform control so that information indicating a time code is displayed on the monitor 13 together with the reference moving image.

[0136] In this case, it is possible to check whether the time code of the video itself captured by camera 12 matches the time code reflected in the video captured by camera 12. This makes it easy to check the synchronization between the capture timing of camera 12 and the display timing of monitor 13.

[0137] <Example of arranging multiple monitors 13> In some cases, actual shooting may be performed with multiple people as subjects, or with people facing various directions. In such cases, it is preferable to place multiple monitors 13 in the shooting space.

[0138] FIG. 14 is a diagram showing another example of the configuration of the imaging system 1, which is a diagram showing an example of the configuration of the imaging system 1 in the case where a plurality of monitors 13 are arranged in the imaging space.

[0139] 14 includes a control device 11, N cameras 12-1 to 12-N, and N monitors 13-1 to 13-N. The cameras 12-1 to 12-N and the monitors 13-1 to 13-N are disposed in the same shooting space.

[0140] Different reference moving images may be displayed on each of the monitors 13-1 to 13-N, or the same reference moving image may be displayed on each of the monitors 13-1 to 13-N.

[0141] Even when a plurality of monitors 13 are provided, the control device 11 controls the timing of the cameras 12-1 to 12-N based on a synchronization signal generated by the control device 11 so that the timing of the images captured by the cameras 12-1 to 12-N and the display timing of the monitors 13-1 to 13-N are synchronized.

[0142] FIG. 15 is a plan view showing a state during actual shooting in the shooting system 1 of FIG. 14 where N=8.

[0143] In the example of FIG. 15, eight cameras 12-1 to 12-8 and eight monitors 13-1 to 13-8 are arranged alternately in a ring shape.

[0144] Monitors 13-1 to 13-8 are respectively arranged with their fronts facing the center, between cameras 12-1 and 12-2, between cameras 12-2 and 12-3, between cameras 12-3 and 12-4, between cameras 12-4 and 12-5, between cameras 12-5 and 12-6, between cameras 12-6 and 12-7, between cameras 12-7 and 12-8, and between cameras 12-8 and 12-1.

[0145] In the examples of Figures 14 and 15, an example is shown in which the same number of monitors 13 as the number of cameras 12 are provided in the photographing system 1, but the number of monitors 13 provided in the photographing system 1 may be different from the number of cameras 12.

[0146] As described above, even when multiple monitors 13 are provided, the control device 11 can perform foreground / background subtraction processing using the actual video and the background video to separate only the subject, for example, a person, from the actual video. At this time, the timing of the cameras 12-1 to 12-N is controlled to be synchronized with the timing of the monitors 13-1 to 13-N to display based on a synchronization signal, thereby making it possible to prevent the reference video of the monitors 13-1 to 13-N that appears in the actual video from being extracted as a subject.

[0147] <Example in which a synchronization signal generating device is provided> In the photographing system 1 of FIG. 7, the synchronization signal generating section 34 is provided inside the control device 11, but the function of the synchronization signal generating section 34 may be realized by a device external to the control device 11.

[0148] FIG. 16 is a diagram showing yet another configuration example of the imaging system 1, in which the function of the synchronization signal generating unit 34 is provided as a separate device.

[0149] The photographing system 1 shown in FIG. 16 is configured by connecting a control device 11A, cameras 12-1 to 12-N, a monitor 13, and a synchronization signal generating device 101 directly or via a network.

[0150] The synchronization signal generating device 101 is provided with the configuration of the synchronization signal generating unit 34 in Fig. 7. That is, the synchronization signal generating device 101 generates synchronization signals in accordance with the control of the control device 11 and supplies them to the cameras 12-1 to 12-N and the monitor 13. The control device 11A has a configuration in which the synchronization signal generating unit 34 is removed from the control device 11 in Fig. 7.

[0151] In this way, a part of the configuration provided in the control device 11 may be provided outside the control device 11.

[0152] <Frame rate conversion example> In each of the above-mentioned embodiments, the frame rate at which camera 12 shoots the actual scene or the background scene is the same as the frame rate at which playback unit 33 plays back the reference video, but the frame rate of the reference video may differ from the frame rate at which camera 12 shoots. In that case, control unit 32 specifies the frame rate to playback unit 33, and playback unit 33 can convert the reference video specified by control unit 32 into the specified frame rate, play it back, and supply it to display unit 52.

[0153] Furthermore, for example, if the lines or lyrics to be displayed on the monitor 13 as reference moving images are composed of one or more still images rather than moving images, the playback unit 33 can convert the still images into moving images having a frame rate specified by the control unit 32 and supply them to the display unit 52.

[0154] <6. Computer configuration examples> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0155] FIG. 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes according to a program.

[0156] A central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are interconnected via a bus 304.

[0157] An input / output interface 305 is further connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0158] The input unit 306 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 307 includes a display, a speaker, an output terminal, etc. The storage unit 308 includes a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0159] In the computer configured as above, the series of processes described above are performed by CPU 301, for example, loading a program stored in storage unit 308 into RAM 303 via input / output interface 305 and bus 304 and executing the program. RAM 303 also stores data and the like necessary for CPU 301 to execute various processes as appropriate.

[0160] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0161] In the computer, the program can be installed in the storage unit 308 via the input / output interface 305 by mounting the removable recording medium 311 in the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0162] In this specification, the steps described in the flowcharts may of course be performed chronologically in the order described, but they do not necessarily have to be processed chronologically and may be executed in parallel or at the required timing, such as when a call is made.

[0163] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0164] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0165] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0166] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0167] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0168] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0169] <Examples of configuration combinations> The present technology can also be configured as follows.

[0170] (1) a control unit that controls synchronization between an image capture timing of an imaging device capturing an image and a display timing of a display device located within an imaging range of the imaging device displaying a first moving image; an image processing unit that performs foreground-background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, The second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized with the second moving image and the third moving image. Control device. (2) the second moving image is a production moving image that is a moving image including a subject and a background, the third moving image is a background moving image that is a moving image including the background, The image processing unit performs the foreground-background subtraction process using the background video image and the actual video image. The control device described in (1). (3) the first moving image is a reference moving image to be used as a reference when shooting the actual moving image, The control unit further performs control to display the reference video on the display device. The control device described in (2). (4) The reference video is a video read from a predetermined storage unit and played back. The control device described in (3). (5) a playback unit that plays back the reference video stored in the predetermined storage unit, The playback unit decodes the reference video image prior to a display timing of the display device. The control device described in (4). (6) The reference video is a video captured by the imaging device during the shooting of the actual video. The control device described in (3). (7) The control unit causes the display device to display the same reference video at the first timing and the second timing. The control device according to any one of (3) to (6). (8) The control unit, when shooting the actual moving image and the background moving image, causes the display device to display the same frame images of the reference moving image at the same frame numbers of the actual moving image and the background moving image. The control device described in (7) above. (9) The control unit generates a synchronization signal, and causes the imaging device to capture an image and the display device to display the first moving image at a timing corresponding to a rising edge or a falling edge of the synchronization signal. The control device according to any one of (1) to (8). (10) The control unit performs control to synchronize a capture timing at which the plurality of imaging devices capture images with a display timing at which the plurality of display devices display the first moving image. The control device according to any one of (1) to (9). (11) The control unit causes the plurality of display devices to respectively display different first moving images. The control device described in (10). (12) The control unit further performs control to display information indicating a time code on the display device together with the first moving image. The control device according to any one of (1) to (11). (13) The apparatus further includes a synchronization signal generating unit that generates the synchronization signal based on the control of the control unit. The control device described in (9). (14) The image processing unit generates difference data by performing foreground / background difference processing between the second moving image and the third moving image captured by the imaging device, and further generates 3D model data based on the difference data. The control device according to any one of (1) to (13). (15) A plurality of imaging devices; At least one display device; a control unit that performs control to synchronize a capture timing of the plurality of imaging devices to capture images with a display timing of the display device located within a capture range of at least one of the plurality of imaging devices to display a first moving image; an image processing unit that performs foreground-background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; Equipped with The second video and the third video both include at least a part of the first video. Shooting system. (16) capturing images using a plurality of imaging devices by synchronizing a capture timing of the imaging devices with a display timing of a display device located within a capture range of at least one of the plurality of imaging devices to display a first moving image; generating difference data by performing foreground / background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; 3. Generating 3D model data based on the differential data How to generate 3D model data. [Explanation of symbols]

[0171] 11 control device, 12-1 to 12-N cameras, 13 monitor, 31 operation unit, 32 control unit, 33 playback unit, 34 synchronization signal generation unit, 35 memory control unit, 36 memory unit, 37 image processing unit, 41 playback data storage unit, 42 shooting data storage unit, 43 foreground background difference processing unit, 44 3D model data generation unit, 51 shooting unit, 52 display unit, 101 synchronization signal generation device

Claims

1. a control unit that controls synchronization between an image capture timing of an imaging device capturing an image and a display timing of a display device located within an imaging range of the imaging device displaying a first moving image; an image processing unit that performs foreground-background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, The second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized with the second moving image and the third moving image. Control device.

2. the second moving image is a production moving image that is a moving image including a subject and a background, the third moving image is a background moving image that is a moving image including the background, The image processing unit performs the foreground-background subtraction process using the background video image and the actual video image. The control device according to claim 1 .

3. the first moving image is a reference moving image to be used as a reference when shooting the actual moving image, The control unit further performs control to display the reference video on the display device. The control device according to claim 2.

4. The reference video is a video read from a predetermined storage unit and played back. The control device according to claim 3.

5. a playback unit that plays back the reference video stored in the predetermined storage unit, The playback unit decodes the reference video image prior to a display timing of the display device. The control device according to claim 4.

6. The reference video is a video captured by the imaging device during the shooting of the actual video. The control device according to claim 3.

7. The control unit causes the display device to display the same reference video at the first timing and the second timing. The control device according to claim 3.

8. The control unit, when shooting the actual moving image and the background moving image, causes the display device to display the same frame images of the reference moving image at the same frame numbers of the actual moving image and the background moving image. The control device according to claim 7.

9. The control unit generates a synchronization signal, and causes the imaging device to capture an image and the display device to display the first moving image at a timing corresponding to a rising edge or a falling edge of the synchronization signal. The control device according to claim 1 .

10. The control unit performs control to synchronize a capture timing at which the plurality of imaging devices capture images with a display timing at which the plurality of display devices display the first moving image. The control device according to claim 1 .

11. The control unit causes the plurality of display devices to respectively display different first moving images. The control device according to claim 10.

12. The control unit further performs control to display information indicating a time code on the display device together with the first moving image. The control device according to claim 1 .

13. The apparatus further includes a synchronization signal generating unit that generates the synchronization signal based on the control of the control unit. The control device according to claim 9.

14. The image processing unit generates difference data by performing foreground / background difference processing between the second moving image and the third moving image captured by the imaging device, and further generates 3D model data based on the difference data. The control device according to claim 1 .

15. A plurality of imaging devices; At least one display device; a control unit that performs control to synchronize a capture timing of the plurality of imaging devices to capture images with a display timing of the display device located within a capture range of at least one of the plurality of imaging devices to display a first moving image; an image processing unit that performs foreground-background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; Equipped with The second video and the third video both include at least a portion of the first video. Shooting system.

16. capturing images using a plurality of imaging devices by synchronizing a capture timing of the imaging devices with a display timing of a display device located within a capture range of at least one of the plurality of imaging devices to display a first moving image; generating difference data by performing foreground / background subtraction processing based on a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; Generate 3D model data based on the difference data A method for generating 3D model data.

Images