Information processing device, control method for information processing device, and program

Abstract

To provide an information processing device capable of applying all lens aberration corrections which are applicable to a motion picture file before CG composition.SOLUTION: A PC 100 performs CG composition on an acquired motion picture file. The PC 100 applies a lens aberration correction which removes a lens aberration to the motion picture file before performing the CG composition. The PC 100 switches whether to use a setting value of the lens aberration correction which is set in imaging of the motion picture file. When using the setting value of the lens aberration correction, the PC 100 applies a non-used lens aberration correction which is identified based on the setting value of the lens aberration correction to the motion picture file, whereas when not using the setting value of the lens aberration correction, the PC applies all the lens aberration corrections which are applicable to the motion picture fille.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In the computer graphics (CG) compositing workflow for virtual production, when compositing live-action footage with CG, lens aberration correction is performed to remove lens aberrations such as peripheral illumination aberration and distortion from the live-action footage before compositing. Since there is no lens aberration whatsoever on the CG side, it is considered desirable for all lens aberration to be removed from the live-action footage before compositing with CG. Furthermore, after CG compositing, the lens aberration of the live-action footage is added to the CG composite result to give the CG a live-action feel.

[0003] In recent years, digital cameras capable of recording RAW video files have appeared, and RAW video files are also used in virtual production because they offer a high degree of freedom in post-shooting video processing. Imaging devices are known that, when recording a RAW video file, record lens aberration correction data for correcting aberrations in the lens attached to the digital camera in association with the RAW video file (see, for example, Patent Document 1). When using a RAW video file for CG compositing, the lens aberration correction data can be obtained from metadata in the RAW video file before the CG compositing, and the lens aberration correction data can be used to remove lens aberration before the CG compositing.

[0004] With digital cameras, users can set which types of lens aberrations to correct (remove) when shooting using menu settings, etc. For example, when recording video, it is possible to set it so that peripheral illumination correction is applied but distortion correction is not applied. When recording video files such as MP4, digital cameras record the resulting video file with lens aberration correction applied according to the settings at the time of shooting. On the other hand, when recording RAW video files, digital cameras generally do not apply lens aberration correction, but instead record the lens aberration correction settings set at the time of shooting in the metadata of the RAW video file. This makes it possible to apply lens aberration correction according to the settings at the time of shooting when developing the RAW video file.

[0005] In virtual production workflows, video files are sometimes cut and edited before CG compositing. At this time, the video file may be converted to a different file format, such as OpenEXR. When a video file is converted to a different file format, the metadata stored within the video file at the time of shooting is often not carried over to the converted file. For this reason, before cutting and editing, users use a camera manufacturer's application to output the metadata within the video file as a metadata file, which is a separate file from the video file. By preparing such a metadata file, lens aberration correction can be applied to each frame image of the video file, which has been cut and converted to a different file format, according to the settings at the time of shooting, before CG compositing. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-23063 Summary of the Invention [Problem to be solved by the invention]

[0007] RAW video files require development processing to display. In addition to camera manufacturer applications, general editing systems may also support RAW video development from various camera manufacturers. For example, Blackmagic Design's DaVinci Resolve supports RAW video development from various camera manufacturers. However, when developing RAW video files, editing systems may not be able to interpret the metadata file output by the camera manufacturer's application and may not be able to apply lens aberration correction according to the lens aberration correction settings in the metadata file. As a result, even if a user specifies a lens aberration correction during shooting, the resulting image developed by the editing system may not include this or any other lens aberration corrections. This results in a discrepancy between the lens aberration correction settings in the metadata file and the applied lens aberration correction in the video file output by the editing system, resulting in a video file before CG compositing in which all lens aberrations cannot be removed.

[0008] As an example, we will describe a case where CG compositing is performed using a RAW video file recorded according to shooting settings that do not apply distortion correction but do apply other lens aberration corrections, and a metadata file output from the RAW video file. In the development process for displaying this RAW video file, lens aberration corrections other than distortion correction are applied according to the shooting settings contained in the metadata file. Furthermore, by applying the unapplied distortion corrections to this developed video file according to the shooting settings, all lens aberrations in this video file are removed before CG compositing. However, when an editing system performs development processing to display this RAW video file, the editing system cannot interpret the metadata file output from the RAW video file and therefore does not apply any lens aberration corrections to this RAW video file. Even if the unapplied distortion corrections are applied to this developed video file according to the shooting settings contained in the metadata file, lens aberration corrections other than distortion correction are not applied. In other words, some lens aberrations in this video file are not removed before CG compositing. As described above, the conventional technology has a problem in that it is not possible to apply all applicable lens aberration corrections to a moving image file before CG synthesis.

[0009] An object of the present invention is to provide an information processing device, a control method for an information processing device, and a program that can apply all applicable lens aberration corrections to a moving image file before CG composition. [Means for solving the problem]

[0010] In order to achieve the above object, the information processing device of the present invention is an information processing device that performs CG synthesis on an acquired video file, and is equipped with a means for applying lens aberration correction to the video file to remove lens aberration before performing the CG synthesis, and a means for switching whether to use a setting value of the lens aberration correction that was set when the video file was shot, and is characterized in that when the setting value is used, the applying means applies to the video file an unapplied lens aberration correction identified based on the setting value, and when the setting value is not used, applies all lens aberration corrections that can be applied to the video file. [Effects of the Invention]

[0011] According to the present invention, all applicable lens aberration corrections can be applied to a video file before CG synthesis. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram schematically illustrating the configuration of a PC as an information processing device according to the present embodiment. [Figure 2] 10 is a flowchart showing the procedure of a metadata file output process executed by the PC of FIG. [Figure 3] 3 is a diagram showing an example of the structure of a metadata file output by the metadata file output process of FIG. 2. FIG. [Figure 4] 10 is a flowchart showing the procedure of a CG synthesis control process executed by a PC in the first embodiment. [Figure 5] 10 is a flowchart showing the procedure of a file format conversion process executed by a PC in the second embodiment. [Figure 6] 10 is a flowchart showing the procedure of a CG composition control process executed by a PC according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, a preferred embodiment for carrying out the present invention will be described with reference to the drawings. In this embodiment, a personal computer (hereinafter referred to as "PC") that handles video files will be described as an example of an information processing device. Furthermore, in a digital camera that provides video files to the information processing device, it is possible to individually set ON / OFF the following lens aberration correction settings during shooting: peripheral illumination correction, chromatic aberration of magnification correction, distortion correction, and focus breathing correction. Peripheral illumination correction is a process for correcting a decrease in peripheral illumination, which is a phenomenon in which the corners of an image become dark due to lens characteristics and shooting conditions. Chromatic aberration of magnification correction is a process for correcting chromatic aberration of magnification, which causes coloring to appear at the edges of the periphery of an image due to lens characteristics. Distortion correction is a process for correcting distortion, which causes distortion in an image due to lens characteristics. Focus breathing correction is a process for reducing fluctuations in the angle of view that occur when the focus position changes during video shooting.

[0014] First, an information processing device and a control method thereof according to a first embodiment of the present invention will be described.

[0015] Fig. 1 is a block diagram showing a schematic configuration of a PC 100 as an information processing device according to this embodiment. In Fig. 1, the PC 100 includes a control unit 101, a ROM 102, a RAM 103, an external storage device 104, an operation unit 105, a display unit 106, and a communication unit 107. These are connected to each other via a bus 108. Note that ROM stands for Read Only Memory, and RAM stands for Random Access Memory.

[0016] The control unit 101 is, for example, a central processing unit (CPU) and controls the entire PC 100. The ROM 102 stores programs and parameters that do not require modification. The RAM 103 temporarily stores programs and data supplied from external devices, etc. The external storage device 104 is a storage device such as a hard disk or flash memory provided within the PC 100, or a memory card that is detachable from the PC 100. The operation unit 105 is a mouse, keyboard, touch panel, etc. that accepts user operations. The display unit 106 displays data held by the PC 100 or data supplied to the PC 100. Note that an external display device may be connected to the PC 100 and data may be displayed on that display device. The communication unit 107 communicates data with an external device such as a digital camera.

[0017] For example, the PC 100 acquires a moving image file recorded by a digital camera (hereinafter referred to as a "camera-recorded moving image file") from the digital camera, and stores the acquired moving image file in the external storage device 104. The camera-recorded moving image file is a non-RAW moving image file or a RAW moving image file.

[0018] Non-RAW video files, like MP4 video files, are video files to which lens aberration correction has been applied by the digital camera according to the lens aberration correction settings set by the user when shooting. Non-RAW video files also contain lens aberration correction data for correcting the aberration of the lens attached to the digital camera.

[0019] RAW video files are video files that have not had lens aberration correction applied by the digital camera. RAW video files contain metadata, including the lens aberration correction settings set by the user when shooting and lens aberration correction data. Thus, video files recorded by a camera contain at least lens aberration correction data.

[0020] The PC 100 includes software modules that perform various processes using video files recorded by a camera, such as an editing system, a CG composition system, and a metadata file output tool.

[0021] An editing system performs cut editing and adds transitions to video files. Examples of editing systems include Apple Final Cut Pro, Avid Media Composer, and Adobe Premiere Pro.

[0022] CG compositing systems reproduce the movement of a real camera in a video file using a virtual camera in a 3DCG space, and then compose the CG so that it is synchronized with the movement of the real camera. CG compositing systems estimate the position and orientation of the real camera by extracting and tracking feature points from images, or by inputting the camera position and orientation from a hardware camera tracker (such as Stype RedSpy or Mo-Sys StarTracker). CG compositing systems incorporate plug-ins from camera manufacturers that remove lens aberration from video files before CG compositing. By passing lens aberration correction data for each lens aberration to the plug-in, the CG compositing system can apply the lens aberration correction designed by the camera manufacturer. Examples of CG compositing systems include Foundry Nuke, Adobe After Effects, Autodesk Flame, and Blackmagic Design Fusion.

[0023] The Metadata File Output Tool is an application developed by a camera manufacturer. When a video file recorded with a digital camera supported by the Metadata File Output Tool is loaded, the tool outputs the metadata contained in the video file as a metadata file. The metadata file is a separate file from the video file, such as a text file or XML file. For example, in a virtual production workflow, a video file may be cut and edited before CG compositing. At this time, the video file may be converted to a different file format, such as OpenEXR. When a video file is converted to a different file format, the metadata contained in the video file at the time of shooting is often not carried over to the converted file. For this reason, on the PC100, the Metadata File Output Tool outputs the metadata contained in the video file as a metadata file before the cut and edit process.

[0024] Fig. 2 is a flowchart showing the procedure of the metadata file output process executed by the PC 100 of Fig. 1. The metadata file output process is realized by the control unit 101 reading a program stored in the ROM 102 or the like into the RAM 103 and executing it.

[0025] In FIG. 2, first, the control unit 101 determines whether or not a moving image file recorded by shooting with a digital camera supported by the metadata file output tool has been read into the metadata file output tool (S201).

[0026] If it is determined in S201 that a video file recorded by shooting with a digital camera supported by the metadata file output tool has not been loaded into the metadata file output tool, this process ends. If it is determined in S201 that a video file recorded by shooting with a digital camera supported by the metadata file output tool has been loaded into the metadata file output tool, this process proceeds to S202.

[0027] In S202, the control unit 101 acquires, from the loaded video file, information unique to the video file (hereinafter referred to as "clip information"). The clip information includes the camera name, lens name, resolution, frame rate, color space at the time of shooting, gamma setting, total number of video frames, and lens aberration correction setting values ​​set at the time of shooting. The lens aberration correction setting values ​​set at the time of shooting are either ON or OFF for peripheral illumination correction, magnification chromatic aberration correction, distortion aberration correction, and focus breathing correction. The clip information acquired in S202 is saved in RAM 103.

[0028] Next, in S203, the control unit 101 sets the frame count n to 1. Next, in S204, the control unit 101 acquires frame information set for the frame image corresponding to the frame count n. The frame information includes the focal length, subject distance, aperture value, color temperature, ISO sensitivity, lens aberration correction data for each lens aberration, etc. The frame information acquired in S204 is also saved in the RAM 103.

[0029] Next, in S205, the control unit 101 determines whether the frame count n matches the total number of frames N of the moving image file read in S201.

[0030] If it is determined in S205 that the frame count n does not match the total number of frames N of the video file loaded in S201, the process proceeds to S206. In S206, the control unit 101 adds 1 to the frame count n. Then, the process returns to S204.

[0031] If it is determined in S205 that the frame count n matches the total number N of frames in the video file loaded in S201, the process proceeds to S207. In S207, the control unit 101 saves the clip information and frame information saved in RAM 103, as well as the metadata file 300 shown in Fig. 3, which includes user setting information (described later) in RAM 103, as a file separate from the video file. Thereafter, the process ends.

[0032] Fig. 3 is a diagram showing an example of the structure of a metadata file 300 output by the metadata file output process of Fig. 2. Fig. 3 shows, as an example, the structure of a metadata file 300 in JSON format. Note that the file format of the metadata file 300 is not limited to JSON format, and may be other file formats such as CSV format or TXT format.

[0033] In the metadata file 300, clip information 310 and user setting information 320 are recorded at the beginning, followed by frame information 330 for the number of frames. The user setting information 320 will now be described. The user setting information 320 is composed of an in-point setting 321, an out-point setting 322, and a shooting lens aberration correction setting ignore flag 323. These are items whose values ​​the user can freely set. For example, when cut editing is performed on a video file recorded with a digital camera using an editing system, the user specifies the frame number of the in-point and the frame number of the out-point specified in the cut editing for the in-point setting 321 and the out-point setting 322, respectively. By referencing these values ​​in the CG compositing system, it is possible to associate each frame image of the converted video file with each frame information of the metadata file.

[0034] The lens aberration correction setting ignore flag 323 during shooting is used when determining the lens aberration correction to be applied before CG composition. The lens aberration correction setting ignore flag 323 is set to "false" by default and can be changed by the user to "true." When the lens aberration correction setting ignore flag 323 during shooting is set to "false," the PC 100 determines the lens aberration correction to be applied before CG composition using the lens aberration correction setting value set during shooting. On the other hand, when the lens aberration correction setting ignore flag 323 during shooting is set to "true," the PC 100 applies all applicable lens aberration corrections to determine the lens aberration correction to be applied before CG composition, without using the lens aberration correction setting value set during shooting. Note that all applicable lens aberration corrections include peripheral illumination correction, chromatic aberration of magnification correction, distortion correction, and focus breathing correction.

[0035] Displaying RAW video files requires development processing. In addition to camera manufacturer applications, the editing system of this embodiment also supports RAW video development from various camera manufacturers. However, when developing RAW video files, this editing system may not be able to analyze the metadata file output from the camera manufacturer's application and may not be able to apply lens aberration correction according to the lens aberration correction setting value in the metadata file. As a result, even if a user sets a certain lens aberration correction to be applied during shooting, the development results produced by the editing system do not apply this lens aberration correction or other lens aberration corrections. This results in a discrepancy between the lens aberration correction setting value in the metadata file and the applied state of the lens aberration correction in the video file output by the editing system, resulting in a problem in which all lens aberrations cannot be removed from the video file before CG composition.

[0036] In contrast, in this embodiment, whether or not to use the lens aberration correction setting value set when shooting the video file is switched, and if this setting value is not used, all lens aberration corrections that can be applied to the video file are applied.

[0037] Fig. 4 is a flowchart showing the procedure of the CG composition control process executed by PC 100 in the first embodiment. The CG composition control process of Fig. 4 is realized by control unit 101 reading a program stored in ROM 102 or the like into RAM 103 and executing it. The CG composition control process of Fig. 4 is started when a user inputs an instruction for CG composition into PC 100, and PC 100 starts the CG composition system in accordance with this instruction. When the CG composition system starts, a screen (not shown) for performing editing operations to compose CG is displayed on display unit 106 of PC 100. This screen includes a video display area for displaying frame images specified by the user in a video file and to be composited with CG.

[0038] 4, first, in S401, the control unit 101 determines whether a video file has been loaded into the CG compositing system. Here, the video file loaded into the CG compositing system in S401 is assumed to be a video file that has been developed using either an application or an editing system manufactured by a camera manufacturer, or a video file recorded by a camera. If it is determined that the video file has not been loaded into the CG compositing system, this process returns to S401.

[0039] If it is determined in S401 that a video file has been loaded into the CG compositing system, the process proceeds to S402. In S402, the control unit 101 displays the first frame image of the video file loaded in S401 in the video display area. Note that in S402, if the video file loaded in S401 is a RAW video file recorded by a camera, a development process is applied to the first frame image of the video file. In this development process, lens aberration correction that was set to "ON" when the video file was captured is applied to the first frame image of the video file. Note that if the video file loaded in S401 is a non-RAW video file recorded by a camera, the lens aberration correction that was set to "ON" when the video file was captured has already been applied to each frame image of the video file by the digital camera. Furthermore, if the video file loaded in S401 is a video file that has already been developed using an application provided by the camera manufacturer, the lens aberration correction that was set to "ON" when the video file was captured has already been applied to each frame image of the video file during the development process. Thus, in this embodiment, if the video file loaded in S401 is a video file recorded by a camera or a video file that has been developed using an application made by the camera manufacturer, when S402 is completed, the lens aberration correction that was set to "ON" when the video was shot will have been applied to the frame images displayed in the video display area. On the other hand, if the video file loaded in S401 is a video file that has been developed using an editing system, not only the lens aberration correction that was set to "ON" when the video was shot, but also other lens aberration corrections have not been applied to each frame image of this video file.

[0040] Next, in S403, the control unit 101 determines whether or not lens aberration correction data necessary for lens aberration correction can be acquired from the video file loaded in S401. As described above, the video file loaded in S401 is either a video file that has been developed using either a camera manufacturer's application or editing system, or a video file recorded on a camera. Of these, a video file recorded on a camera includes lens aberration correction data. Therefore, in S403, if the video file loaded in S401 is a video file recorded on a camera, it is determined that the lens aberration correction data necessary for lens aberration correction can be acquired from the video file loaded in S401. In this case, the process proceeds to S408.

[0041] In S408, the control unit 101 performs a first lens aberration removal control process using the lens aberration correction setting value set at the time of shooting. In the first lens aberration removal control process, the control unit 101 acquires the lens aberration correction setting value set at the time of shooting and lens aberration correction data corresponding to the first frame image from the video file. Furthermore, the control unit 101 applies only the lens aberration corrections that were set to "OFF" at the time of shooting, among the applicable lens aberration corrections, to the first frame image of the video file, and displays the processing results in the video display area. For example, if, among peripheral illumination correction, chromatic aberration correction for magnification, distortion correction, and focus breathing correction, only peripheral illumination correction was set to "OFF" and the others were set to "ON" at the time of shooting, only peripheral illumination correction is applied in S408. As described above, when S402 is completed, the lens aberration corrections that were set to "ON" at the time of shooting have been applied to the first frame image of the video file recorded by the camera. In step S408, the lens aberration correction that was set to "OFF" during shooting is further applied to such a frame image, so that all lens aberrations are removed from this frame image. Next, the process proceeds to step S409, which will be described later.

[0042] On the other hand, in S403, if the video file loaded in S401 has been developed using either a camera manufacturer's application or editing system, it is determined that the lens aberration correction data required for lens aberration correction cannot be obtained from the video file loaded in S401. In this case, the process proceeds to S404.

[0043] In S404, the control unit 101 determines whether the metadata file 300 corresponding to the video file loaded in S401 has been loaded into the CG compositing system. If it is determined that the metadata file 300 corresponding to the video file loaded in S401 has not been loaded into the CG compositing system, the process returns to S404.

[0044] If it is determined in S404 that the metadata file 300 corresponding to the video file loaded in S401 has been loaded into the CG compositing system, the process proceeds to S405. In S405, the control unit 101 determines whether the value of the lens aberration correction setting ignore flag 323 during shooting included in the metadata file 300 loaded in S404 is “true.” In this embodiment, as described above, the lens aberration correction setting ignore flag 323 during shooting is set by the user. For example, when loading a video file developed by an editing system that cannot analyze the metadata file 300 into the CG compositing system, the user sets the lens aberration correction setting ignore flag 323 during shooting to “true.” On the other hand, when loading a video file developed by a camera manufacturer's application into the CG compositing system, the user sets the lens aberration correction setting ignore flag 323 to the default value, “false.”

[0045] If it is determined in S405 that the value of the lens aberration correction setting ignore flag 323 during shooting is “true,” the process proceeds to S406. In S406, the control unit 101 performs a second lens aberration removal control process that does not use the lens aberration correction setting value set during shooting. In the second lens aberration removal control process, the control unit 101 acquires lens aberration correction data corresponding to the first frame image from the frame information 330 in the metadata file 300. The control unit 101 also applies all applicable lens aberration corrections to the first frame image of the video file and displays the processing results in the video display area. By performing this control, even if no lens aberration corrections have been applied to the video file during development processing by the editing system, all lens aberration corrections are applied to the video file according to the setting of the lens aberration correction setting ignore flag 323 during shooting. In other words, it is possible to remove all lens aberrations from a video file that has been developed by the editing system. After completing the process of S406, the process proceeds to S409, which will be described later.

[0046] If it is determined in S405 that the value of the lens aberration correction setting ignore flag 323 during shooting is not "true," that is, if the value of the lens aberration correction setting ignore flag 323 during shooting is "false," the process proceeds to S407.

[0047] In S407, the control unit 101 performs a third lens aberration removal control process using the lens aberration correction setting value set at the time of shooting. In the third lens aberration removal control process, the control unit 101 acquires the lens aberration correction setting value set at the time of shooting from the clip information 310 in the metadata file 300 and acquires lens aberration correction data corresponding to the first frame image from the frame information 330 in the metadata file 300. The control unit 101 also applies only the lens aberration correction that was set to "OFF" at the time of shooting to the first frame image of the video file and displays the processing result in the video display area. As described above, the lens aberration correction that was set to "ON" at the time of shooting was applied during development to each frame image of a video file that has been developed using a camera manufacturer's application. In S408, by further applying the lens aberration correction that was set to "OFF" at the time of shooting to such a frame image, all lens aberrations in this frame image are removed. The process then proceeds to S409.

[0048] In S409, the control unit 101 performs CG compositing processing in accordance with instructions received from the user. The live-action video used here has had all applicable lens aberration corrections applied, and all lens aberrations have been removed, making it possible to achieve natural CG compositing. When the CG compositing processing is complete, this process ends.

[0049] In the above-described embodiment, the processing for the first frame image of a video file has been described as an example, but the same processing is also applied to subsequent frame images. For example, if the user selects the second frame image, the same processing as that performed on the first frame image among S406 to S408 is performed on the second frame image. If the video file loaded in S401 is a RAW video file recorded by a camera, development processing is applied to the selected second frame image. In this development processing, lens aberration correction that was set to "ON" when the image was captured is applied to the selected second frame image. In this way, all applicable lens aberration corrections can be applied to the frame image selected by the user before CG composition, thereby eliminating all lens aberrations.

[0050] According to the first embodiment described above, whether or not to use the lens aberration correction setting value set when the moving image file was shot is switched, and if this setting value is not used, all lens aberration corrections that can be applied to the moving image file are applied. In other words, a means is provided for applying all lens aberration corrections that can be applied to the moving image file without using the lens aberration correction setting value set when the moving image file was shot. This makes it possible to deal with cases where the lens aberration correction setting value set when the moving image file was shot does not match the application state of the lens aberration correction in the moving image file output by the editing system. As a result, all lens aberration corrections that can be applied can be applied to the moving image file before CG composition.

[0051] In the first embodiment described above, all of the lens aberration corrections that can be applied are peripheral illumination correction, chromatic aberration of magnification correction, distortion correction, and focus breathing correction. This allows these lens aberration corrections to be applied to video files before CG composition.

[0052] Furthermore, in the first embodiment described above, whether or not to use the lens aberration correction setting value set when the moving image file was captured is switched based on the value of the shooting-time lens aberration correction setting ignore flag 323 set by the user. This allows the user's intention to be reflected in the switching of whether or not to use the lens aberration correction setting value set when the moving image file was captured.

[0053] Next, an information processing device and a control method thereof according to a second embodiment of the present invention will be described.

[0054] The second embodiment is basically the same as the first embodiment in terms of configuration and operation, but differs from the first embodiment in that it switches between using or not using the lens aberration correction setting value that was set when the video file was shot, based on the type of development processing applied. Therefore, a description of the overlapping configuration and operation will be omitted, and the following will describe the different configuration and operation.

[0055] In the first embodiment described above, for example, a user may forget to set the lens aberration correction setting ignore flag 323 to "true" even though the user has loaded into the CG compositing system a moving image file that has been developed by an editing system that cannot analyze the metadata file 300. In such a case, it becomes impossible to apply all applicable lens aberration corrections to the moving image file before CG compositing.

[0056] In contrast to this, in the second embodiment, whether or not to use the lens aberration correction setting value that was set when the moving image file was captured is switched based on the type of development processing that was applied.

[0057] 5 is a flowchart showing the procedure of file format conversion processing executed by PC 100 in the second embodiment. The file format conversion processing of FIG. 5 is realized by control unit 101 reading a program stored in ROM 102 or the like into RAM 103 and executing it. The file format conversion processing of FIG. 5 is started when a user inputs an instruction to PC 100 to edit a RAW video file as preparation for CG compositing, and PC 100 starts up the editing system in accordance with this instruction. When the editing system starts up, a screen (not shown) for editing the RAW video file is displayed on display unit 106 of PC 100. This screen includes a video display area for displaying a frame image specified by the user in the RAW video file and to be edited.

[0058] 5, first, in S501, the control unit 101 determines whether or not a RAW video file has been loaded into the editing system. If it is determined that a RAW video file has not been loaded into the editing system, the process returns to S501. If it is determined that a RAW video file has been loaded into the editing system, the process proceeds to S502.

[0059] In S502, the control unit 101 causes the editing system to perform development processing on the first frame image of the RAW moving image file read in S501, and displays the developed frame image in the moving image display area.

[0060] Next, in S503, the control unit 101 accepts editing operations from the user. User editing operations include setting in and out points for the RAW video file, adjusting white balance, and setting color space and gamma. Some editing systems incorporate libraries provided by camera manufacturers so that development processing can be performed using processes designed by the camera manufacturers. In such editing systems, the user can select either the editing system manufacturer's or the camera manufacturer's development processing to execute. When the camera manufacturer's development processing, which uses the camera manufacturer's library, is selected, the editing system can analyze the metadata in the RAW video file. Therefore, when developing the RAW video file, the editing system can read the lens aberration correction setting value and lens aberration correction data set at the time of shooting from the metadata and apply the lens aberration correction that was set to "ON" at the time of shooting. On the other hand, when the editing system manufacturer's development processing, which does not use the camera manufacturer's library, is selected, the editing system cannot analyze the metadata in the RAW video file. For this reason, when developing RAW video files, this editing system cannot apply not only the lens aberration correction that was set to "ON" at the time of shooting, but also other lens aberration corrections.

[0061] Next, in S504, the control unit 101 determines whether or not an instruction to convert the file format of the RAW video file edited in accordance with the editing operation accepted in S503 into a predetermined format (for example, MP4) has been received. If it is determined that an instruction to convert the file format into a predetermined format has not been received, the process returns to S503. If it is determined that an instruction to convert the file format into a predetermined format has been received, the process proceeds to S505.

[0062] In S505, the control unit 101 performs development processing on the frame images in the in-point / out-point section of the RAW video file using predetermined development settings, and outputs an edited video file including the developed frame images and audio data.

[0063] Next, in S506, the control unit 101 writes information about the type of development processing that was applied to the metadata area of ​​the edited video file output in S505. The information about the type of development processing that was applied is information that indicates whether the applied development processing is a development processing from the editing system manufacturer or the camera manufacturer. When the processing of S506 is completed, this processing ends.

[0064] FIG. 6 is a flowchart showing the procedure of CG composition control processing executed by PC 100 in the second embodiment. Note that the CG composition control processing in FIG. 6 is similar to the CG composition control processing in FIG. 4 described above, and the following will particularly describe the differences from the CG composition control processing in FIG. 4 described above. Like the CG composition control processing in FIG. 4 described above, the CG composition control processing in FIG. 6 is also realized by the control unit 101 reading a program stored in ROM 102 or the like into RAM 103 and executing it. Also, like the CG composition control processing in FIG. 4 described above, the CG composition control processing in FIG. 6 is started when the user inputs an instruction to perform CG composition into PC 100, and PC 100 starts the CG composition system in accordance with this instruction.

[0065] 6, first, S601 and S602, which are the same processes as S401 and S402 described above, are performed. Next, in S603, the control unit 101 determines whether or not lens aberration correction data required for lens aberration correction can be acquired from the video file loaded in S601. Note that the determination method in S603 is the same as the determination method in S403 described above.

[0066] If it is determined in S603 that the lens aberration correction data required for lens aberration correction can be acquired from the video file loaded in S601, the process proceeds to S608, which is the same process as S408 described above. Next, the process proceeds to S609, which is the same process as S409 described above, and then the process ends.

[0067] If it is determined in S603 that the lens aberration correction data required for lens aberration correction cannot be acquired from the video file loaded in S601, the process proceeds to S604. In S604, the control unit 101 determines whether the metadata file 300 corresponding to the video file loaded in S601 has been loaded into the CG compositing system. If it is determined that the metadata file 300 corresponding to the video file loaded in S601 has not been loaded into the CG compositing system, the process returns to S604.

[0068] If it is determined in S604 that the metadata file 300 corresponding to the video file read in S601 has been read into the CG synthesis system, the process proceeds to S605.

[0069] In S605, the control unit 101 determines whether the information about the type of development processing that was applied, which information was read from the metadata of the moving image file that was read in S601, indicates the development processing of the camera manufacturer.

[0070] If it is determined in S605 that the information read from the metadata of the video file loaded in S601 does not indicate the camera manufacturer's development process, i.e., if the read information indicates the editing system manufacturer's development process, the process proceeds to S606. In S606, the control unit 101 performs a second lens aberration removal control process that does not use the lens aberration correction setting value set at the time of shooting, as in S406 described above. As described above, the editing system manufacturer's development process does not apply not only the lens aberration correction set to "ON" at the time of shooting, but also other lens aberration corrections to the video file. By performing the second lens aberration removal control process on such a video file, all lens aberration corrections are applied. In other words, it is possible to remove all lens aberrations from the video file that has been developed by the editing system manufacturer. The process then proceeds to S609.

[0071] If it is determined in S605 that the information read from the metadata of the video file loaded in S601 indicates the camera manufacturer's development process, the process proceeds to S607. In S607, the control unit 101 performs a third lens aberration removal control process using the lens aberration correction setting value set at the time of shooting, similar to S407 described above. In the camera manufacturer's development process, as described above, the lens aberration correction that was set to "ON" at the time of shooting is applied to the video file. By performing the third lens aberration removal control process on such a video file, the lens aberration correction that was set to "OFF" at the time of shooting is also applied. In other words, it is possible to remove all lens aberrations from the video file that has been processed by the camera manufacturer. The process then proceeds to S609.

[0072] In the above-described embodiment, the processing for the first frame image of a video file has been described as an example, but the same processing is also applied to subsequent frame images. For example, if the user selects the second frame image, the same processing as that performed on the first frame image among S606 to S608 is performed on the second frame image. Note that if the video file loaded in S601 is a RAW video file recorded by a camera, development processing is applied to the selected second frame image. In this development processing, lens aberration correction that was set to "ON" at the time of shooting is applied to the selected second frame image. In this way, all applicable lens aberration corrections can be applied to the frame image selected by the user before CG composition, thereby eliminating all lens aberrations.

[0073] In the second embodiment described above, if the information read from the metadata of a video file indicates the camera manufacturer's development process, it is determined that the lens aberration correction setting value set when the video file was shot will be used. If the information read from the metadata of a video file does not indicate the camera manufacturer's development process, it is determined that the lens aberration correction setting value set when the video file was shot will not be used. This makes it possible to appropriately switch whether to use the lens aberration correction setting value set when the video file was shot, depending on the type of development process applied.

[0074] In the second embodiment described above, information about the type of development processing applied is written to the metadata of a video file edited by an editing system, but the present invention is not limited to this configuration. For example, the name of the editing system may be written to the metadata of a video file edited by the editing system. This makes it possible to appropriately switch whether to use the lens aberration correction setting value that was set when the video file was shot, depending on the type of application that performed the development processing.

[0075] Furthermore, in the second embodiment described above, when the editing system applies a camera manufacturer's development process, it may be configured to embed a digital watermark in the development result to identify that the development process is that of the camera manufacturer. In such a configuration, whether the development process applied to the video file is that of the camera manufacturer is determined based on the presence or absence of the digital watermark. For example, if a plug-in incorporated in the CG compositing system extracts the digital watermark, it is determined that the development process applied to the video file is that of the camera manufacturer. In this case, in the CG compositing control process described above in FIG. 6, the process of S607 is performed on the frame image selected by the user. On the other hand, if the plug-in incorporated in the CG compositing system does not extract the digital watermark, it is determined that the development process applied to the video file is not that of the camera manufacturer, but that of the editing system manufacturer. In this case, in the CG compositing control process described above in FIG. 6, the process of S606 is performed on the frame image selected by the user. Even with this configuration, the same effects as those of the above-described embodiment can be achieved.

[0076] In this embodiment, a plug-in incorporated in the CG compositing system may be controlled to display a UI. This UI allows the user to set whether to determine the lens aberration correction to be applied using the lens aberration correction setting value set at the time of shooting, or to apply all applicable lens aberration corrections without using the lens aberration correction setting value set at the time of shooting. Furthermore, this UI may also allow the user to select the lens aberration correction to be applied.

[0077] In CG compositing, color space and gamma settings of live-action footage are sometimes converted to match those of the CG. In editing systems, however, the color space and gamma settings may be changed by the user when converting video files to a different format, resulting in a discrepancy in color space and gamma information stored in the metadata file. In such cases, color space and gamma information can be obtained from the converted video itself, rather than from the color space and gamma settings in the metadata file, and color space and gamma conversion can be performed when applying lens aberration correction. If the color space and gamma settings configured in the editing system are not standard color space and gamma settings but are proprietary to the editing system, a warning message can be displayed, as the camera manufacturer's plug-in cannot perform the conversion.

[0078] Furthermore, the present invention can be applied not only to cases where lens aberrations are removed before CG composition, but also to cases where lens aberrations are added to the CG composition results. When lens aberrations are added to the CG composition results, the determinations in S405 and S605 are performed. For example, if the result in S405 is NO, the lens aberrations to be added are determined using the lens aberration correction setting value set at the time of shooting. If the result in S405 is YES, all lens aberrations that can be added without using the lens aberration correction setting value set at the time of shooting are added. Also, if the result in S605 is YES, the lens aberrations to be added are determined using the lens aberration correction setting value set at the time of shooting. If the result in S605 is NO, all lens aberrations that can be added without using the lens aberration correction setting value set at the time of shooting are added.

[0079] As a specific example, we will consider a RAW video file recorded using a digital camera with peripheral illumination correction set to "OFF" and distortion correction set to "ON." Note that when adding lens aberration to the CG composite results, there is little need to add chromatic aberration of magnification and focus breathing as visual effects. Therefore, the following discussion focuses on the ON / OFF setting of lens aberration addition processing for peripheral illumination and distortion. The metadata for the RAW video file contains information indicating peripheral illumination correction: OFF and distortion correction: ON. When this RAW video file is input into the metadata file output tool, the clip information 310 in the metadata file 300 contains information indicating the lens aberration correction settings set at the time of shooting: peripheral illumination correction: OFF and distortion correction: ON. When adding lens aberration using this metadata file 300, if the lens aberration correction setting ignore flag 323 in the metadata file 300 is set to "false," peripheral illumination aberration is added to the CG composite results. On the other hand, if the lens aberration correction setting ignore flag 323 during shooting is "true," peripheral illumination aberration and distortion aberration are added to the CG composition result. By processing in this way, it is possible to control whether the CG composition result is restored to the state of the lens aberration correction setting value set during shooting, or to a state in which all lens aberrations are added.

[0080] In this embodiment, the CG synthesis system has been described as being capable of analyzing metadata included in a RAW video file. However, if the CG synthesis system cannot analyze metadata, it cannot identify the lens aberration correction settings set at the time of shooting. As a result, it is not possible to apply all applicable lens aberration corrections to the RAW video file before CG synthesis. In response to this, the PC 100 may determine whether the loaded video file was recorded by a specific model of imaging device supported by the CG synthesis system. If the loaded video file was recorded by a specific model of imaging device supported by the CG synthesis system, it may decide to use the lens aberration correction settings set at the time of shooting and perform the third lens aberration removal control process described above. On the other hand, if the loaded video file was not recorded by a specific model of imaging device supported by the CG synthesis system, it may decide not to use the lens aberration correction settings set at the time of shooting and perform the second lens aberration removal control process described above. By controlling in this manner, it is possible to apply all applicable lens aberration corrections to the RAW video file before CG synthesis, even if the CG synthesis system cannot analyze the metadata of the loaded video file.

[0081] Furthermore, some lenses attached to digital cameras have significant lens distortion, so distortion correction is always applied. In such cases, distortion correction cannot be switched on or off in the digital camera's menu settings, and distortion correction is always "ON" during shooting. If lens aberration is added to the CG composite result of a video recorded using a lens that always applies distortion correction, there is a concern that excessive distortion may be added as a visual effect. For this reason, even if the lens aberration correction setting ignore flag 323 in the metadata file 300 is "true," distortion may not be applied.

[0082] While the present invention has been described in detail above based on preferred embodiments, it is not limited to these specific embodiments, and various modifications within the spirit and scope of the present invention are also encompassed by the present invention. Parts of the above-described embodiments may be combined as appropriate. Furthermore, the present invention also encompasses a case in which a software program implementing the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program, either directly from a recording medium or via wired or wireless communication, and the program is then executed. Therefore, the program code itself supplied to and installed on a computer to implement the functional processing of the present invention also embodies the present invention. In other words, the computer program itself for implementing the functional processing of the present invention is also encompassed by the present invention. In this case, the program may take any form, such as object code, a program executed by an interpreter, or script data supplied to an OS, as long as it has the program functionality. Examples of recording media for providing the program include magnetic recording media such as hard disks and magnetic tapes, optical / magneto-optical storage media, and nonvolatile semiconductor memory. Another conceivable method for providing the program is to store the computer program implementing the present invention on a server on a computer network, and then download the computer program to a connected client computer.

[0083] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An information processing device that performs CG synthesis on an acquired video file, comprising: a means for applying lens aberration correction to the video file to remove lens aberration before performing the CG synthesis; and a means for switching whether to use a setting value for lens aberration correction that was set when the video file was shot; wherein the applying means, when using the setting value, applies to the video file an unapplied lens aberration correction identified based on the setting value; and when not using the setting value, applies all lens aberration corrections that can be applied to the video file. (Configuration 2) The information processing device described in Configuration 1, wherein the applying means, when using the setting value, adds a lens aberration determined based on the setting value from among multiple lens aberrations that can be added to the CG-combined video file, and when not using the setting value, adds all lens aberrations that can be added to the CG-combined video file. (Configuration 3) The information processing device according to configuration 1 or 2, wherein all of the applicable lens aberration corrections are peripheral illumination correction, chromatic aberration of magnification correction, distortion correction, and focus breathing correction. (Configuration 4) An information processing device described in any one of configurations 1 to 3, further comprising a means for reading a metadata file generated based on metadata included in the video file, wherein the switching means switches whether to use the lens aberration correction setting value set when the video file was shot based on flag information included in the metadata file, and the flag information is flag information indicating whether to use the lens aberration correction setting value set when the video file was shot, and is set by a user operating the information processing device. (Configuration 5) An information processing device described in any one of configurations 1 to 4, further comprising a means for acquiring information indicating the type of development processing applied to the video file from the video file, wherein the switching means decides to use the lens aberration correction setting value set when the video file was shot if the acquired information indicates development processing using a library provided by the manufacturer of the imaging device that recorded the video file, and decides not to use the lens aberration correction setting value set when the video file was shot if the acquired information does not indicate development processing using the library. (Configuration 6) An information processing device described in any one of configurations 1 to 5, further comprising a means for acquiring information indicating a system that output the video file from the video file, wherein the switching means decides to use the lens aberration correction setting value that was set when the video file was shot if the acquired information indicates a system provided by the manufacturer of the imaging device that recorded the video file, and decides not to use the lens aberration correction setting value that was set when the video file was shot if the acquired information does not indicate a system provided by the manufacturer of the imaging device that recorded the video file. (Configuration 7) An information processing device described in any one of configurations 1 to 6, further comprising a means for determining whether the video file contains an electronic watermark for identifying that the development process was performed using a library provided by the manufacturer of the imaging device that recorded the video file, wherein the switching means decides to use the lens aberration correction setting value that was set when the video file was shot if the video file contains the electronic watermark, and decides not to use the lens aberration correction setting value that was set when the video file was shot if the video file does not contain the electronic watermark. (Configuration 8) An information processing device described in any one of configurations 1 to 7, further comprising a means for determining whether the video file is a video file recorded by a specific imaging device, wherein the switching means, if it is determined that the video file is a video file recorded by the specific imaging device, decides to use the lens aberration correction setting value that was set when the video file was shot, and, if it is determined that the video file is not a video file recorded by the specific imaging device, decides not to use the lens aberration correction setting value that was set when the video file was shot. (Configuration 9) The information processing device according to any one of configurations 1 to 8, wherein the moving image file is a moving image file obtained by converting the file format of a RAW moving image file recorded by an imaging device into another file format. [Explanation of symbols]

[0084] 100 PC 101 Control section 300 Metadata File

Claims

1. An information processing device that performs CG compositing on an acquired video file, a means for applying lens aberration correction to the moving image file to remove lens aberration before the CG compositing; a means for switching whether or not to use a lens aberration correction setting value that was set when the moving image file was shot; The information processing device is characterized in that the applying means, when using the setting value, applies to the video file any unapplied lens aberration correction identified based on the setting value, and when not using the setting value, applies to the video file all lens aberration corrections that can be applied.

2. The information processing device according to claim 1, characterized in that, when the setting value is used, the applying means adds a lens aberration determined based on the setting value from among a plurality of lens aberrations that can be added to the CG-composited moving image file, and when the setting value is not used, the applying means adds all lens aberrations that can be added to the CG-composited moving image file.

3. 2. The information processing apparatus according to claim 1, wherein the all of the applicable lens aberration corrections are peripheral illumination correction, chromatic aberration of magnification correction, distortion correction, and focus breathing correction.

4. The video processing device further includes a reading unit for reading a metadata file generated based on metadata included in the video file, the switching means switches whether or not to use a setting value of lens aberration correction that was set when the moving image file was shot, based on flag information included in the metadata file; The information processing device according to claim 1, characterized in that the flag information indicates whether or not to use a lens aberration correction setting value that was set when the video file was shot, and is set by a user operating the information processing device.

5. further comprising means for acquiring, from the moving image file, information indicating the type of development processing applied to the moving image file; The information processing device according to claim 1, characterized in that the switching means decides to use the lens aberration correction setting values ​​that were set when the video file was shot if the acquired information indicates development processing using a library provided by the manufacturer of the imaging device that recorded the video file, and decides not to use the lens aberration correction setting values ​​that were set when the video file was shot if the acquired information does not indicate development processing using the library.

6. The video file output device further includes a means for acquiring information indicating a system that outputs the video file from the video file, The information processing device according to claim 1, characterized in that the switching means decides to use the lens aberration correction setting value that was set when the video file was shot if the acquired information indicates a system provided by the manufacturer of the imaging device that recorded the video file, and decides not to use the lens aberration correction setting value that was set when the video file was shot if the acquired information does not indicate a system provided by the manufacturer of the imaging device that recorded the video file.

7. The image processing device further includes a means for determining whether the moving image file contains a digital watermark for identifying that the moving image file has been developed using a library provided by the manufacturer of the imaging device that recorded the moving image file, The information processing device described in claim 1, characterized in that the switching means decides to use the lens aberration correction setting value set when the video file was shot if the digital watermark is included in the video file, and decides not to use the lens aberration correction setting value set when the video file was shot if the digital watermark is not included in the video file.

8. The video file may further include a determination unit for determining whether the video file is a video file recorded by a specific imaging device, The information processing device according to claim 1, characterized in that the switching means decides to use the lens aberration correction setting value that was set when the video file was shot if it is determined that the video file was a video file recorded by the specific imaging device, and decides not to use the lens aberration correction setting value that was set when the video file was shot if it is determined that the video file was not a video file recorded by the specific imaging device.

9. 2. The information processing apparatus according to claim 1, wherein the moving image file is a moving image file obtained by converting a file format of a RAW moving image file recorded by an imaging device into another file format.

10. A control method for an information processing device that performs CG compositing on an acquired video file, comprising: applying lens aberration correction to the video file to remove lens aberration before performing the CG compositing; a step of switching whether or not to use a lens aberration correction setting value that was set when the moving image file was captured; A control method for an information processing device, characterized in that the applying step, when using the setting value, applies to the video file any unapplied lens aberration correction identified based on the setting value, and when not using the setting value, applies to the video file all lens aberration corrections that can be applied.

11. A program for causing a computer to execute a control method for an information processing device that performs CG compositing on an acquired video file, The control method for the information processing device includes: applying lens aberration correction to the video file to remove lens aberration before performing the CG compositing; a step of switching whether or not to use a lens aberration correction setting value that was set when the moving image file was captured; The program is characterized in that the applying step, when the setting value is used, applies to the video file any unapplied lens aberration corrections identified based on the setting value, and when the setting value is not used, applies to the video file all lens aberration corrections that can be applied.

Images