Video processing system, video processing method, and program

Abstract

The present disclosure provides a video processing system, a video processing method, and a program with which it is possible to generate a composite video having a higher degree of completion. At the time of imaging, a first video processing unit generates and displays a simple composite video by performing first video processing for simply synthesizing CG with video and creates a metafile including meta-information used in synthesizing the CG with the video. After imaging, a second video processing unit generates a highly accurate composite video by performing second video processing for synthesizing CG with video at high accuracy using a background in which the simple composite video generated by using the metafile is displayed. The present technology can be applied, for example, to a video processing system that synthesizes two-dimensional or three-dimensional CG with video.

Description

Image processing system, image processing method, and program

[0001] The present disclosure relates to a video processing system, a video processing method, and a program, and more particularly to a video processing system, a video processing method, and a program that are capable of generating a more complete composite video.

[0002] Conventionally, for example, a technique has been developed for tracking the movement of feature points in an image in the time direction in order to synthesize two-dimensional or three-dimensional CG (Computer Graphics) in accordance with the movement of the image.

[0003] For example, Patent Document 1 discloses a synthesis method for synthesizing three-dimensional CG in real time, in which positions on an image are identified by inter-frame tracking of feature points from pre-captured moving image data.

[0004] Japanese Patent Application Laid-Open No. 2008-15576

[0005] However, in the past, when shooting, users would have to shoot while checking the image that has not been composited with CG, so they would have to imagine what the composite image would look like when composited with CG. As a result, the composite image that has been composited with CG in the later image processing may have been shot with camerawork that differs from the user's imagination, which reduces the quality of the composite image.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to generate a more complete composite video.

[0007] An image processing system according to one aspect of the present disclosure includes a first image processing unit that, during shooting, generates and displays a simplified composite image by performing a first image processing operation that simply combines CG with the image, and creates a metafile containing meta information used to combine the CG with the image, and a second image processing unit that, after shooting, generates a highly accurate composite image by performing a second image processing operation that combines CG with the image with high precision in the background while the simplified composite image generated using the metafile is displayed.

[0008] An image processing method or program according to one aspect of the present disclosure includes, during shooting, performing first image processing to simply combine CG with the image to generate and display a simple composite image, creating a metafile containing meta information used to combine the CG with the image, and, after shooting, performing second image processing to highly accurately combine CG with the image in the background while the simple composite image generated using the metafile is displayed, thereby generating a highly accurate composite image.

[0009] In one aspect of the present disclosure, at the time of shooting, a first image processing is performed to simply combine CG with the image, thereby generating and displaying a simple composite image, and a metafile containing meta information used to combine the CG with the image is created, and after shooting, a high-precision composite image is generated by performing a second image processing to highly precisely combine CG with the image in the background while the simple composite image generated using the metafile is displayed.

[0010] FIG. 1 is a diagram illustrating an example of use of a video processing system to which the present technology is applied. FIG. 2 is a block diagram illustrating an example configuration of an embodiment of a video processing system. FIG. 3 is a flowchart illustrating video processing for generating a simple composite video during shooting. FIG. 4 is a flowchart illustrating video processing for generating a highly accurate composite video during main line video processing. FIG. 4 is a diagram illustrating an example of use of a video processing system using 2DCG. FIG. 5 is a diagram illustrating an example of a UI image for instructing the execution of video processing during shooting. FIG. 6 is a diagram illustrating an example of a UI image for instructing the execution of video processing during main line video processing. FIG. 7 is a block diagram illustrating an example configuration of an embodiment of a computer to which the present technology is applied.

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Example of Use of Video Processing System> An example of use of a video processing system to which the present technology is applied will be described with reference to FIG.

[0013] The upper part of Fig. 1 shows an example of image capture using the image processing system 11. A camera 21 captures an image of a subject 12 and transmits the captured image to a smartphone 22. Various communication standards, such as HDMI (High-Definition Multimedia Interface) (registered trademark), SDI (Serial Digital Interface), USB (Universal Serial Bus) (registered trademark), and wireless LAN (Local Area Network), can be used to transmit the image from the camera 21 to the smartphone 22. The smartphone 22 then performs image processing (tracking, compositing, and rendering) to simply composite a 3DCG model onto the subject 12 captured in the image captured by the camera 21, and displays the resulting simplified composite image.

[0014] Therefore, by using the video processing system 11, the user can check in real time on the smartphone 22 a simplified composite image in which a 3DCG model is simply composited onto the subject 12 during shooting. This allows the user to shoot with camerawork that allows for framing that takes into account the composition of the 3DCG model onto the subject 12. Furthermore, when reviewing the shot video, the user can check the simplified composite image in which a 3DCG model is simply composited onto the subject 12, allowing the user to consider at that time whether or not it is necessary to re-shoot the subject 12.

[0015] Then, in the image processing system 11, after the camera 21 has finished capturing the image of the subject 12, the image can be processed using the personal computer 23. That is, the image processing during capture only simply synthesizes a 3DCG model, and it is necessary to synthesize a 3DCG model with higher accuracy.

[0016] The lower part of Fig. 1 shows an example of main line video processing using video processing system 11, in which personal computer 23 acquires a main line video file from camera 21 via recording media 24 and acquires a 3DCG metafile by communicating with smartphone 22. The main line video file is a file containing video without 3DCG composited therein, and the 3DCG metafile is a file containing various meta information used to composite a 3DCG model onto subject 12 through simple video processing performed on smartphone 22. Then, while playing back the simple composite video using the 3DCG metafile, personal computer 23 performs video processing in the background to composite a 3DCG model onto subject 12 with high precision, thereby generating a higher-quality composite video.

[0017] In this way, by using the video processing system 11, the user can have the personal computer 23 generate a more complete composite video during main line video processing immediately after the camera 21 has finished capturing the video. This allows the user to more quickly begin editing and releasing the video.

[0018] While FIG. 1 illustrates the video processing for superimposing a 3DCG model on the subject 12, the present technology is not limited to video processing for superimposing a 3DCG model on the subject 12. For example, the present technology can also be applied to video processing for superimposing a 3DCG model on a space other than the subject in the video. That is, video processing can be performed to superimpose a 3DCG model of a refrigerator, bed, or other object into a space in a video of a room containing only a desk and chair. Furthermore, when performing such video processing, the present technology tracks the camera 21, so that the position and orientation of the 3DCG model can be superimposed to match the movement of the camera 21 and the space in the video.

[0019] 1 , when the smartphone 22 is fixed to the camera 21 and the image processing system 11 is used, the smartphone 22 may perform simple image processing using the output of the six-axis sensor provided in the camera 21, or may perform simple image processing using the output of the six-axis sensor provided in the smartphone 22. On the other hand, when the image processing system 11 is used without fixing the smartphone 22 to the camera 21, it is necessary to supply the output of the six-axis sensor provided in the camera 21 to the smartphone 22, and the smartphone 22 performs simple image processing using the output of the six-axis sensor of the camera 21.

[0020] <Configuration Example of Video Processing System> FIG. 2 is a block diagram showing a configuration example of an embodiment of a video processing system to which the present technology is applied.

[0021] As shown in FIG. 2, the video processing system 11 includes a camera 21, a smartphone 22, and a personal computer 23, and is configured so that a 3DCG model is imported from a model storage unit 25 to the smartphone 22 and the personal computer 23.

[0022] The camera 21 comprises a posture recognition unit 31, a photographing unit 32, a time code generation unit 33, a data transmission unit , a display unit 35, and a recording media drive .

[0023] The attitude recognition unit 31 recognizes the attitude of the camera 21 and sequentially supplies attitude information indicating the attitude to the data transmission unit 34. For example, the attitude recognition unit 31 can recognize the attitude of the camera 21 based on six-axis data indicating six-axis acceleration or angular velocity detected by a six-axis sensor provided in the camera 21.

[0024] The photographing unit 32 photographs the subject 12 using an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and supplies the image obtained by the photographing to the data transmission unit 34, the display unit 35, and the recording media drive 36.

[0025] The time code generating unit 33 generates a time code according to the frame rate of the video captured by the imaging unit 32 and supplies it to the data transmitting unit 34 and the recording media drive 36 .

[0026] The data transmission unit 34 adds the time code supplied from the time code generation unit 33 to each frame of the video supplied from the imaging unit 32. Furthermore, the data transmission unit 34 adds, to each frame of the video supplied from the imaging unit 32, posture information of the camera 21 sequentially supplied from the posture recognition unit 31, which posture information is obtained at the timing corresponding to each frame. Then, the data transmission unit 34 sequentially transmits the video to which the time code and the posture information of the camera 21 have been added to the smartphone 22.

[0027] The display unit 35 displays a live view image, which is a real-time image being captured by the image capturing unit 32 .

[0028] The recording media drive 36 adds a time code supplied from the time code generation unit 33 to each frame of the video captured by the imaging unit 32, and records a main line video file containing that video on the recording media 24 (FIG. 1). The main line video file may also include a proxy file in which the resolution of the video captured by the imaging unit 32 is reduced, as necessary. Note that the main line video file may include orientation information of the camera 21; for example, if orientation information of the camera 21 or smartphone 22 is included in a 3DCG metafile (described later), the orientation information of the camera 21 does not need to be included in the main line video file.

[0029] The smartphone 22 is configured to include a posture recognition unit 41 , a simple tracking unit 42 , a simple synthesis processing unit 43 , a simple rendering unit 44 , a display unit 45 , and a metafile creation unit 46 .

[0030] The attitude recognition unit 41 recognizes the attitude of the smartphone 22 and sequentially supplies attitude information indicating the attitude to the simple tracking unit 42 and the metafile creation unit 46. For example, the attitude recognition unit 41 can recognize the attitude of the smartphone 22 based on six-axis data indicating six-axis acceleration or angular velocity detected by a six-axis sensor provided in the smartphone 22.

[0031] The simplified tracking unit 42 simply tracks the self-position of the camera 21 based on the posture information supplied from the posture recognition unit 41 and feature points extracted from the video (including the subject 12 and all of the scenery in the video other than the subject 12). For example, when the smartphone 22 is fixed to the camera 21 and the video processing system 11 is used (see FIG. 1), the simplified tracking unit 42 can use the posture information of the smartphone 22 supplied from the posture recognition unit 41 and the posture information of the camera 21 added to the video transmitted from the camera 21. On the other hand, when the video processing system 11 is used without fixing the smartphone 22 to the camera 21, the simplified tracking unit 42 can use only the posture information of the camera 21 added to the video transmitted from the camera 21. The simplified tracking unit 42 then supplies tracking information indicating the results of the simplified tracking of the self-position of the camera 21 (e.g., information indicating the simplified position, orientation, posture, etc. of the camera 21) to the simplified synthesis processing unit 43.

[0032] The simple composition processing unit 43 performs a composition process to simply compose the 3DCG model imported from the model storage unit 25 onto the video transmitted from the camera 21 based on the tracking information supplied from the simple tracking unit 42. For example, the simple composition processing unit 43 determines the simple position, orientation, posture, etc. of the 3DCG model to be composed onto the video so as to correspond to the scenery in the video and the position, orientation, posture, etc. of the subject 12. The simple composition processing unit 43 then generates simple 3DCG model composition information that indicates the simple position, orientation, posture, etc. of the 3DCG model and supplies the information to the simple rendering unit 44.

[0033] The simple rendering unit 44 positions the 3DCG model based on the simple 3DCG model synthesis information supplied from the simple synthesis processing unit 43, sets a virtual camera corresponding to the camera 21 based on the tracking information, and uses the virtual camera to simply render the 3DCG model. The simple rendering unit 44 then generates a simple synthesized video by superimposing the image of the 3DCG model obtained by rendering on each frame of the video transmitted from the camera 21, and supplies the generated video to the display unit 45.

[0034] The display unit 45 displays the simple composite image supplied from the simple rendering unit 44, i.e., an image obtained by simply combining the image captured by the camera 21 with an image of a 3DCG model rendered by a virtual camera corresponding to the camera 21.

[0035] The metafile creation unit 46 creates a 3DCG metafile containing various meta information used to composite a 3DCG model onto video through simple video processing performed on the smartphone 22, and transmits the created 3DCG metafile to the personal computer 23. For example, the 3DCG metafile contains at least simple 3DCG model composite information (XYZ coordinates indicating simple position information of the 3DCG model) with a time code added for each frame, and posture information (XYZ coordinates) with the time code added. The 3DCG metafile also contains the model name of the 3DCG model, simple position information (XYZ coordinates) of the 3DCG model at the time of shooting start, zoom / focus / iris (plus gain / white balance) information, etc.

[0036] The personal computer 23 comprises a recording media drive 51 , a synchronization processing section 52 , a tracking section 53 , a synthesis processing section 54 , a rendering section 55 , and a display section 56 .

[0037] When the recording media 24 (FIG. 1) on which the main line video file is recorded is set by the recording media drive 36 of the camera 21, the recording media drive 51 reads the main line video file from the recording media 24 and supplies it to the synchronization processing unit 52.

[0038] The synchronization processing unit 52 performs synchronization processing to synchronize the 3DCG metafile transmitted from the smartphone 22 with the video included in the main line video file supplied from the recording media drive 51 according to their respective time codes. The synchronization processing unit 52 then supplies the synchronized main line video file to the tracking unit 53 and the rendering unit 55, and supplies the synchronized 3DCG metafile to the composition processing unit 54 and the rendering unit 55.

[0039] The tracking unit 53 tracks the self-position of the camera 21 with high accuracy based on the posture information included in the 3DCG metafile supplied from the synchronization processing unit 52 and feature points extracted from the video included in the main line video file supplied from the synchronization processing unit 52. Then, the tracking unit 53 supplies the synthesis processing unit 54 with tracking information indicating the results of tracking the self-position of the camera 21 with high accuracy (for example, information indicating the highly accurate position, orientation, posture, etc. of the camera 21).

[0040] The composition processing unit 54 performs composition processing to composite with high precision the 3DCG model imported from the model storage unit 25 onto the video included in the main line video file supplied from the synchronization processing unit 52, based on the tracking information supplied from the tracking unit 53. For example, the composition processing unit 54 determines with high precision the position, orientation, posture, etc. of the 3DCG model to be composited onto the video so as to correspond to the scenery in the video and the position, orientation, posture, etc. of the subject 12. The composition processing unit 54 then generates 3DCG model composition information indicating the high-precision position, orientation, posture, etc. of the 3DCG model and supplies the information to the rendering unit 55.

[0041] The rendering unit 55 uses the simple 3DCG model synthesis information contained in the 3DCG metafile supplied from the synchronization processing unit 52 to generate a simple synthesized image in the same manner as the simple rendering unit 44, and supplies it to and displays it on the display unit 56. In this way, tracking by the tracking unit 53 and synthesis processing by the synthesis processing unit 54 are performed in the background while the simple synthesized image generated by the rendering unit 55 is displayed on the display unit 56.

[0042] Furthermore, in the background where the simple composite video is displayed on the display unit 56, the rendering unit 55 positions the 3DCG model based on the 3DCG model composition information supplied from the composition processing unit 54, sets a virtual camera corresponding to the camera 21 based on the tracking information, and uses the virtual camera to render the 3DCG model with high precision. The rendering unit 55 then superimposes the image of the 3DCG model obtained by rendering on each frame of the video included in the main line video file supplied from the synchronization processing unit 52 to generate a high-precision composite video and supplies it to the display unit 56.

[0043] The display unit 56 displays the simple composite image supplied from the rendering unit 55, and thereafter, when the simple composite image is supplied with a high-precision composite image from the display unit 56, the display unit 56 displays the high-precision composite image.

[0044] The video processing system 11 is configured as described above, and the user can shoot with camerawork that allows for more appropriate framing while checking the simple composite video displayed on the smartphone 22. Furthermore, during main line video processing, the user can edit the video while realistically imagining what the composite video will look like by checking the simple composite video displayed on the personal computer 23. Then, a higher-quality composite video can be generated in the background while the simple composite video is displayed on the personal computer 23. This allows the video processing system 11 to generate a more complete composite video.

[0045] 3 and 4, a description will be given of the image processing executed in the image processing system 11. Here, a processing example will be described in which the orientation information of the camera 21 is used, and not the orientation information of the smartphone 22.

[0046] 3 is a flowchart illustrating image processing for generating a simple composite image during image capture. For example, the processing starts when the user performs an operation to perform image processing in which the camera 21 and the smartphone 22 are linked.

[0047] In step S11, in the camera 21, the photographing unit 32 starts photographing the subject 12, the posture recognition unit 31 starts recognizing the posture of the camera 21, and the time code generation unit 33 starts generating a time code. Then, the data transmission unit 34 sequentially transmits the images to which the time code and posture information of the camera 21 have been added to the smartphone 22.

[0048] In step S12, the simple tracking unit 42 tracks the self-position of the camera 21 based on the attitude information of the camera 21 added to the video transmitted in step S11, feature points extracted from the video transmitted in step S11, etc. Then, the simple tracking unit 42 supplies tracking information indicating the result of the simple tracking of the self-position of the camera 21 to the simple synthesis processing unit 43.

[0049] In step S13, the simple synthesis processing unit 43 performs synthesis processing to simply synthesize a 3DCG model onto the video transmitted in step S11 based on the tracking information supplied in step S12, generates simple 3DCG model synthesis information, and supplies it to the simple rendering unit 44.

[0050] In step S14, the simple rendering unit 44 positions the 3DCG model based on the simple 3DCG model synthesis information supplied in step S13, sets a virtual camera corresponding to the camera 21 based on the tracking information supplied in step S12, and performs simple rendering of the 3DCG model using the virtual camera. The simple rendering unit 44 then superimposes the image of the 3DCG model obtained by rendering on the video transmitted from the camera 21 to generate a simple synthesized video and supplies the generated video to the display unit 45.

[0051] In step S15, the display unit 45 displays the simplified composite image supplied in step S14.

[0052] In step S16, the metafile creation unit 46 creates a 3DCG metafile that includes at least, for example, the posture information of the camera 21 added to the video transmitted in step S11 and the simple 3DCG model synthesis information generated in step S13, and the like, to which a time code has been added.

[0053] After the process of step S16, the process returns to step S11, and the same process is repeated thereafter until, for example, the user performs an operation to end the video processing.

[0054] 4 is a flowchart illustrating the image processing for generating a high-precision composite image during main line image processing. For example, the user inserts the recording medium 24 on which the main line image file is recorded into the personal computer 23, connects the smartphone 22 and the personal computer 23, and then performs an operation to start image processing.

[0055] In step S21, the metafile creation unit 46 in the smartphone 22 transmits the 3DCG metafile created in step S16 of FIG.

[0056] In step S 22 , the recording media drive 51 reads the main line video file from the recording media 24 and supplies it to the synchronization processing unit 52 .

[0057] In step S23, the synchronization processing unit 52 performs synchronization processing to synchronize the 3DCG metafile transmitted in step S21 with the video included in the main line video file supplied in step S22 according to their respective time codes. The synchronization processing unit 52 then supplies the synchronized main line video file to the tracking unit 53 and the rendering unit 55, and supplies the synchronized 3DCG metafile to the composition processing unit 54 and the rendering unit 55.

[0058] In step S24, the rendering unit 55 uses the simple 3DCG model synthesis information contained in the 3DCG metafile supplied in step S23 to generate a simple synthesis image in the same manner as the simple rendering unit 44, and supplies it to the display unit 56 for display.

[0059] Meanwhile, in parallel with the process of step S24, the processes of steps S25 to S27 are carried out.

[0060] In step S25, the tracking unit 53 tracks the self-position of the camera 21 with high accuracy based on the posture information included in the 3DCG metafile supplied in step S23 and feature points extracted from the image included in the main line image file supplied in step S23. Then, the tracking unit 53 supplies tracking information indicating the result of tracking the self-position of the camera 21 with high accuracy to the synthesis processing unit 54.

[0061] In step S26, the synthesis processing unit 54 performs synthesis processing to synthesize a 3DCG model with high precision onto the image contained in the main line image file supplied in step S23 based on the tracking information supplied in step S25, and generates 3DCG model synthesis information and supplies it to the rendering unit 55.

[0062] In step S27, the rendering unit 55 positions the 3DCG model based on the 3DCG model synthesis information supplied in step S26, sets a virtual camera corresponding to camera 21 based on the tracking information supplied in step S25, and uses the virtual camera to render the 3DCG model with high precision. The rendering unit 55 then superimposes the image of the 3DCG model obtained by rendering on each frame of the video included in the main line video file supplied from the synchronization processing unit 52 to generate a high-precision synthesized video and supplies the generated video to the display unit 56.

[0063] In step S28, the display unit 56 displays the high-precision composite image supplied in step S27 in place of the simple composite image displayed in step S24.

[0064] After the process of step S28, the process returns to step S21, and the same process is repeated thereafter until, for example, the user performs an operation to end the video processing.

[0065] By performing the above-described image processing, the image processing system 11 can generate a more complete composite image by displaying a simple composite image during shooting and generating a composite image in the background that displays the simple composite image during main line image processing.

[0066] <Example of Use of Video Processing System> With reference to FIG. 5, an example of use of a 2D model in a video processing system will be described.

[0067] 1, the image processing system 11 can synthesize a 3DCG model of a subject captured in an image during image capture and main line image processing. Furthermore, the image processing system 11 can synthesize a 2D model of a subject in the same way as a 3DCG model.

[0068] As shown in the upper part of Fig. 5, a camera 21 photographs a subject 13 while transmitting the image acquired by the photograph to a smartphone 22, and the smartphone 22 performs image processing to simply composite a 2D model onto the subject 13 shown in the image, and displays the resulting simplified composite image. Note that, although a mosaic image is shown as the 2D model in the example shown in Fig. 5, other images such as stamps can also be used as the 2D model.

[0069] Thereafter, as shown in the lower part of Fig. 5, the personal computer 23 acquires the main line video file from the camera 21 via the recording medium 24 and acquires a 2D metafile by communicating with the smartphone 22. The 2D metafile is a file containing various meta information used to superimpose a 2D model onto the subject 12 through simple video processing performed on the smartphone 22. The personal computer 23 then plays back the simple composite video using the 2D metafile, while performing video processing in the background to superimpose the 2D model onto the subject 12 with high precision, thereby generating a higher-quality composite video. Of course, in addition to superimposing a 2D model onto the subject 13, a 2D model may also be superimposed on a space other than the subject 13 in the video.

[0070] In this way, by using the video processing system 11, it is possible to generate a more complete composite image on the personal computer 23, similar to the example of using the 3DCG model described above with reference to Figure 1.

[0071] For example, the 2D metafile contains at least simple 2D model synthesis information (XYZ coordinates indicating simple position information of the 2D model) with a time code added for each frame. The 2D metafile can also contain the model name of the 2D model, zoom / focus / iris (plus gain / white balance) information, etc. The orientation of the 2D model does not need to be changed in accordance with the movement of the camera 21.

[0072] <Display Example of UI Image> A UI (User Interface) image for instructing the execution of video processing using the video processing system 11 will be described with reference to FIGS. 6 and 7. FIG.

[0073] FIG. 6 is a diagram showing an example of a UI image for instructing the execution of video processing during shooting.

[0074] 6A shows a UI image displayed on the display unit 35 of the camera 21, displaying a "Yes" GUI (Graphical User Interface) button and a "No" GUI button along with the message "Do you want to link posture information and video to perform simple composition?". FIG. 6B shows a UI image displayed on the display unit 45 of the smartphone 22, displaying a "Yes" GUI button and a "No" GUI button along with the message "Do you want to receive posture information, video, and 2D / 3DCG models to perform simple composition?".

[0075] For example, when the user displays UI images as shown in the figure on the display units 35 and 45 and performs a user operation on each of the "Yes" GUI buttons, transmission of posture information and video from the camera 21 to the smartphone 22 begins. Then, the smartphone 22 begins receiving the posture information and video, and imports the 2D / 3DCG model from the model storage unit 25. Note that the import of the 2D / 3DCG model may be performed in advance.

[0076] FIG. 7 is a diagram showing an example of a UI image for instructing execution of video processing during main line video processing.

[0077] 7A shows a UI image displayed on the display unit 35 of the camera 21, displaying a message asking "Do you want to send video to edit the main line video?" along with a "Yes" GUI button and a "No" GUI button. FIG. 7B shows a UI image displayed on the display unit 45 of the smartphone 22, displaying a message asking "Do you want to send a 2D / 3DCG metafile to edit the main line video?" along with a "Yes" GUI button and a "No" GUI button. FIG. 7C shows a UI image displayed on the display unit 56 of the personal computer 23, displaying a message asking "Do you want to receive a 2D / 3DCG metafile, video, and 2D / 3DCG model to edit the main line video?" along with a "Yes" GUI button and a "No" GUI button.

[0078] For example, when a user displays UI images as shown on display units 35, 45, and 56 and operates the respective "Yes" GUI buttons, transmission of video from camera 21 to personal computer 23 begins, and transmission of a 2D / 3DCG metafile from smartphone 22 to personal computer 23 begins. Then, personal computer 23 begins receiving the video and 2D / 3DCG metafile, and imports the 2D / 3DCG model from model storage unit 25. Note that the import of the 2D / 3DCG model may be performed in advance.

[0079] In the video processing system 11, such a UI image can be used to instruct the execution of video processing.

[0080] In this embodiment, CG compositing processing refers to compositing CG with video captured by the camera 21 in subsequent editing. The CG also needs to move in accordance with the movement of the camera 21. Tracking refers to tracking feature points in the video captured by the camera 21, such as objects in the video, to perform 2D tracking or 3D camera tracking. 2D tracking refers to tracking objects in the video using feature points and an acceleration sensor in the video captured by the camera 21, and is used, for example, when it is desired to have a mosaic constantly follow a person in the video. 3D camera tracking refers to analyzing how the camera 21 was moved to capture the video based on an acceleration sensor, feature points, etc., and generating a virtual camera. Generally, 3D camera tracking links the CG and the virtual camera, allowing the composited CG to move in the same way in accordance with the movement of the video, enabling a natural composition.

[0081] The gyro included in the camera 21 may be a 6-axis or 8-axis sensor, or may be another sensor. Instead of the smartphone 22, a tablet may be used, or an external gyro sensor may be used in combination with an external monitor. Furthermore, if the camera 21 has the function of performing 3D tracking and composition processing, there is no need to attach an external smartphone 22, and the above-described processing of the smartphone 22 can be performed by the camera 21. Instead of generating a simple composite video simultaneously with capturing an image by the camera 21, a simple composite video may be generated using images already captured by the camera 21.

[0082] Furthermore, if the smartphone 22 has processing capabilities equivalent to those of the personal computer 23, the smartphone 22 may perform main line image processing, in which case there is no need to transmit the main line image file and the 3DCG metafile to the personal computer 23. Furthermore, if the camera 21 has processing capabilities equivalent to those of the smartphone 22 and the personal computer 23, all image processing may be performed within the camera 21, in which case there is no need to transmit images from the camera 21 to the outside.

[0083] <Example of Computer Configuration> Next, the above-described series of processes (video processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0084] FIG. 8 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0085] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0086] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0087] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0088] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0089] When a user inputs a command via the input / output interface 110 by operating the input unit 107, the CPU 102 executes a program stored in a read-only memory (ROM) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored on a hard disk 105 into a random access memory (RAM) 104 and executes the program.

[0090] As a result, the CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 102 outputs the processing results from the output unit 106 via the input / output interface 110, transmits them from the communication unit 108, or records them on the hard disk 105, as necessary.

[0091] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0092] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0093] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0094] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0095] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0096] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0097] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0098] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0099] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0100] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0101] <Examples of Combinations of Configurations> The present technology can also be configured as follows: (1) An image processing system comprising: a first image processing unit that, during shooting, generates and displays a simplified composite image by performing first image processing that simply combines CG (Computer Graphics) with an image, and creates a metafile including meta information used to combine the CG with the image, and a second image processing unit that, after shooting, generates a highly accurate composite image by performing second image processing that combines CG with the image with high precision in the background while the simplified composite image generated using the metafile is displayed. (2) The image processing system according to (1), further comprising: a camera that performs shooting, wherein the first image processing unit and the second image processing unit are operated based on attitude information indicating the attitude of the camera. (3) The video processing system according to (2), wherein the first video processing unit has: a simplified tracking unit that simply tracks the camera based on the shooting direction of the camera identified based on the posture information and feature points extracted from the video, a simplified compositing processing unit that generates simplified compositing information required to composite the CG onto the video based on a result of the tracking, and a simplified rendering unit that arranges the CG based on the simplified compositing information, sets a virtual camera corresponding to the camera identified based on the posture information, and simply renders the CG using the virtual camera to generate the simplified composite video. (4) The camera has a time code generation unit that generates a time code that identifies a frame of the video captured by the camera, and the second video processing unit has a synchronization processing unit that synchronizes the metafile and the video in accordance with the time code added to the metafile and the time code added to each frame of the video.(5) The image processing system according to (4), wherein the second image processing unit has: a tracking unit that tracks the camera with high accuracy based on the shooting direction of the camera specified based on the posture information and feature points extracted from the image; a composition processing unit that generates composition information necessary for compositing the CG onto the image based on a result of the tracking; and a rendering unit that arranges the CG based on the composition information, sets a virtual camera corresponding to the camera specified based on the posture information, and renders the CG with high accuracy using the virtual camera to generate the composite image. (6) The image processing system according to (5), wherein tracking by the tracking unit, generation of the composition information by the composition processing unit, and generation of the composite image by the rendering unit are performed in the background while the rendering unit generates and displays the simple composite image using the simple composition information included in the metafile. (7) A video processing method comprising: an video processing system, during shooting, performing first video processing to simply composite CG (Computer Graphics) onto video to generate and display a simplified composite video, and creating a metafile containing meta information used to composite the CG onto the video, and after shooting, performing second video processing to composite CG with high precision onto the video in a background where the simplified composite video generated using the metafile is displayed, to generate a highly accurate composite video. (8) A program for causing a computer of the video processing system to execute video processing, including: performing first video processing to simply composite CG (Computer Graphics) onto video during shooting, to generate and display a simplified composite video, and creating a metafile containing meta information used to composite the CG onto the video, and after shooting, performing second video processing to composite CG with high precision onto the video in a background where the simplified composite video generated using the metafile is displayed, to generate a highly accurate composite video.

[0102] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0103] REFERENCE SIGNS LIST 11 Image processing system, 12 and 13 Subject, 21 Camera, 22 Smartphone, 23 Personal computer, 24 Recording media, 25 Model storage unit, 31 Posture recognition unit, 32 Photography unit, 33 Time code generation unit, 34 Data transmission unit, 35 Display unit, 36 Recording media drive, 41 Posture recognition unit, 42 Simple tracking unit, 43 Simple composition processing unit, 44 Simple rendering unit, 45 Display unit, 46 Metafile creation unit, 51 Recording media drive, 52 Synchronization processing unit, 53 Tracking unit, 54 Composition processing unit, 55 Rendering unit, 56 Display unit

Claims

1. An image processing system comprising: a first image processing unit that, during shooting, performs a first image processing to simply combine CG (Computer Graphics) with the image to generate and display a simplified composite image, and creates a metafile including meta information used to combine the CG with the image; and after shooting, a second image processing unit that, in the background while the simplified composite image generated using the metafile is displayed, performs a second image processing to highly accurately combine CG with the image, to generate a highly accurate composite image.

2. The image processing system according to claim 1, further comprising a camera for taking images, wherein the first image processing unit and the second image processing unit are operated based on posture information indicating the posture of the camera.

3. The image processing system of claim 2, wherein the first image processing unit has: a simplified tracking unit that simply tracks the camera based on the shooting direction of the camera identified based on the posture information and feature points extracted from the image; a simplified synthesis processing unit that generates simplified synthesis information required to synthesize the CG onto the image based on the tracking results; and a simplified rendering unit that positions the CG based on the simplified synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and simply renders the CG using the virtual camera to generate the simple synthesized image.

4. The video processing system of claim 3, wherein the camera has a time code generation unit that generates a time code that identifies a frame of the video captured by the camera, and the second video processing unit has a synchronization processing unit that synchronizes the metafile and the video in accordance with the time code added to the metafile and the time code added to each frame of the video.

5. The image processing system of claim 4, wherein the second image processing unit has: a tracking unit that tracks the camera with high accuracy based on the shooting direction of the camera identified based on the posture information and feature points extracted from the image; a synthesis processing unit that generates synthesis information required to synthesize the CG onto the image based on the tracking results; and a rendering unit that positions the CG based on the synthesis information, sets a virtual camera corresponding to the camera identified based on the posture information, and renders the CG with high accuracy using the virtual camera to generate the synthetic image.

6. The image processing system of claim 5, wherein tracking by the tracking unit, generation of the synthesis information by the synthesis processing unit, and generation of the synthetic image by the rendering unit are performed in the background while the rendering unit generates and displays the simple synthetic image using the simple synthesis information contained in the metafile.

7. A video processing method comprising: a video processing system performing, during shooting, a first video processing to simply combine CG (Computer Graphics) with the video to generate and display a simplified composite image, and creating a metafile containing meta information used to combine the CG with the video; and after shooting, performing a second video processing to highly accurately combine CG with the video in the background while the simplified composite image generated using the metafile is displayed, thereby generating a highly accurate composite image.

8. A program for causing a computer of an image processing system to execute image processing including: performing a first image processing to simply combine CG (Computer Graphics) with the image during shooting to generate and display a simplified composite image, and creating a metafile containing meta information used to combine the CG with the image; and performing a second image processing to highly precisely combine CG with the image after shooting in the background while the simplified composite image generated using the metafile is displayed, thereby generating a highly precise composite image.

Images