Information processing device, control method for information processing device, and program

Abstract

To provide an information processing device capable of reducing a sense of discontinuity in a boundary of a first clip part and a second clip part while giving an atmosphere of real imaging to the second clip part in a coupled clip which is obtained by coupling a second clip composed of images having no lens aberration to a first clip composed of motion pictures obtained by imaging.SOLUTION: An information processing device 100 acquires a frame image to which a lens aberration is not applied from a coupled clip 205, acquires a frame number of the frame image, and acquires a value of a focal distance which is preset correspondingly to the frame number from virtual camera information 501. The information processing device 100 acquires lens meta data corresponding to a value of a focal distance which is closest to the value of the focal distance from lens aberration meta data 300 of a camera material clip 202 and uses the lens meta data to apply the lens aberration to the frame image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] Information processing devices such as PCs that perform various processes on images captured by a camera are known. In the information processing device, for example, image processing is performed to add an image free of lens aberrations such as peripheral light falloff and distortion, such as a CG image, to a live-action image captured by a camera. In this image processing, a first process is performed to remove lens aberration from the live-action image, a second process is performed to add a CG image to the live-action image from which the lens aberration has been removed, and the lens aberration removed in the first process is applied to the composite image obtained by the second process. In this way, the same lens aberration as in the live-action image is also applied to the CG image added to the live-action image, thereby creating a natural composite image that gives the CG image a live-action feel.

[0003] Furthermore, information processing devices perform a process of chronologically combining a video recorded by a camera with another video or the like. For example, when combining two different videos in chronological order, the metadata of each of these videos is compared. If the comparison shows that the shooting mode information contained in the metadata is the same, the metadata is carried over to a combined clip obtained by combining the two videos (see, for example, Patent Document 1). For example, by using information about lens aberration contained in this metadata, it is possible to apply the lens aberration of one of the two combined videos to the other video, thereby generating a combined clip in which the sense of discontinuity at the transition between the two videos is reduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-092583 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, when data other than video captured in the same shooting mode as the video recorded by a camera, such as a CG clip composed of CG images, is combined in chronological order, metadata is not inherited by the combined clip. Therefore, it is impossible to obtain lens aberration information for the camera material clip portion composed of video captured by a camera. As a result, it is impossible to perform processes such as removing lens aberration from the camera material clip portion or applying lens aberration to the entire combined clip, which would impart a realistic feel to the CG clip portion while reducing the sense of discontinuity at the transition between the camera material clip portion and the CG clip portion. Furthermore, even if metadata for the video constituting the camera material clip portion were inherited, this metadata only contains information about the lens aberration of the camera material clip portion, making it impossible to apply lens aberration to the CG clip portion so as to avoid a sense of discontinuity at the transition to the camera material clip portion. In other words, conventional techniques have been unable to impart a realistic feel to the CG clip portion while reducing the sense of discontinuity at the transition between the camera material clip portion and the CG clip portion.

[0006] The present invention aims to provide an information processing device, a control method for an information processing device, and a program that can reduce the sense of discontinuity at the transition between the first clip portion and the second clip portion in a combined clip obtained by combining a first clip obtained by removing lens aberration from a video generated by shooting with a shooting device with a second clip consisting of images without lens aberration, while giving the second clip portion an atmosphere of real action. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing device of the present invention is provided, which is configured to: and, and the second clip consisting of images without lens aberrations. Arranged chronologically An information processing device that generates a combined clip includes: means for reading metadata of the first clip, the metadata of the first clip having a plurality of lens metadata corresponding to a plurality of frame images that constitute the first clip, the plurality of lens metadata indicating amounts of lens aberration correction, recorded in association with shooting setting values ​​used in the shooting; means for acquiring a frame image to which lens aberration has not been applied from the plurality of frame images that constitute the combined clip, and acquiring a frame number of the frame image in the combined clip; means for acquiring a first shooting setting value that is set in advance in association with the acquired frame number; and means for acquiring lens metadata from the metadata of the first clip that corresponds to a second shooting setting value that is closest to the acquired first shooting setting value, and performing processing to apply lens aberration to the acquired frame image using the lens metadata. [Effects of the Invention]

[0008] According to the present invention, in a combined clip obtained by combining a first clip obtained by removing lens aberration from a video generated by shooting with a shooting device with a second clip consisting of images without lens aberration, it is possible to give the second clip portion an atmosphere of real life while reducing the sense of discontinuity at the transition between the first clip portion and the second clip portion. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a schematic configuration of an information processing device according to an embodiment of the present invention. [Figure 2] 1. FIG. 4 is a diagram for explaining a video editing process performed by the information processing device of FIG. [Figure 3] FIG. 3 is a diagram showing an example of lens aberration metadata for the camera footage clip of FIG. 2. [Figure 4] 6 is a flowchart showing the procedure of lens aberration application processing executed by the information processing device according to the first embodiment. [Figure 5] 5 is a diagram for explaining the configuration of a combined clip and the configuration of virtual camera information used in the lens aberration application processing of FIG. 4. [Figure 6] 5 is a diagram for explaining a combined clip to which lens aberration has been applied, which is output in the lens aberration application processing of FIG. 4. FIG. [Figure 7] 10 is a flowchart showing the procedure of lens aberration application processing executed by an information processing device according to a second embodiment. [Figure 8] 8 is a diagram for explaining the configuration of a combined clip used in the lens aberration application processing of FIG. 7. FIG. [Figure 9] 8 is a diagram for explaining a combined clip to which lens aberration has been applied, which is output in the lens aberration application processing of FIG. 7. FIG. [Figure 10] 11 is a flowchart showing the procedure of lens aberration application processing executed by an information processing device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the present embodiment, a personal computer (hereinafter referred to as "PC") that handles video files will be described as an example of an information processing device.

[0011] First, an information processing device and a control method thereof according to a first embodiment of the present invention will be described.

[0012] Fig. 1 is a block diagram showing a schematic configuration of an information processing device 100 according to an embodiment of the present invention. In Fig. 1, the information processing device 100 includes a control unit 101, a ROM 102, a RAM 103, a storage device 104, an operation unit 105, and a display unit 106. These are connected to each other via a bus 107. Note that ROM stands for Read Only Memory, and RAM stands for Random Access Memory.

[0013] The control unit 101 is, for example, a CPU (Central Processing Unit) and controls the entire information processing device 100. The ROM 102 stores programs and parameters that do not require modification. For example, the ROM 102 stores a program for an application for image editing (hereinafter referred to as an "image editing application"). The RAM 103 temporarily stores programs and data acquired from an external device or the like. The storage device 104 is, for example, a hard disk or flash memory. Alternatively, the storage device 104 may be an external storage device such as an optical disk such as a floppy disk (FD) or a compact disk (CD), a magnetic or optical card, an IC card, or a memory card that is detachable from the information processing device 100. The operation unit 105 is composed of buttons and a touch panel for inputting data and accepts user operations. The display unit 106 displays data held by the information processing device 100, data acquired from an external device, and the like.

[0014] On the PC 100, a user loads a video captured by a digital camera (not shown) (photographing device) into an image editing application, and edits the video using the image editing application. In the editing process, the user arranges, for example, a CG material clip 201 composed of a CG image and a camera material clip 202 composed of the video in this order on a timeline on the image editing application, as shown in FIG. 2(a). Note that lens aberration occurs in the camera material clip 202. For example, distortion aberration occurs in the camera material clip 202, as shown in image 204. Note that, in this embodiment, a case where distortion aberration occurs will be described as an example of lens aberration, but lens aberration is not limited to distortion aberration. Examples of lens aberration include peripheral shading. On the other hand, the CG material clip 201 does not suffer from lens aberration because it was not captured with a photographing device equipped with a lens (see, for example, image 203).

[0015] Next, the PC 100 performs lens aberration removal processing. In the lens aberration removal processing, the PC 100 removes lens aberration from the camera material clip 202 using lens aberration metadata 300 of Fig. 3, which includes shooting setting values ​​and the like used in shooting the video that constitutes the camera material clip 202. As a result, as shown in Fig. 2(b), neither the camera material clip 202 nor the CG material clip 201 becomes in a state where lens aberration has occurred.

[0016] FIG. 3 shows an example of lens aberration metadata 300 for the camera footage clip 202 in FIG. 2. The lens aberration metadata 300 is composed of a common information section 301, an additional information section 302, and a frame-by-frame information section 303. The lens aberration metadata 300 is in text format, with data for each frame recorded along with the frame number, separated by commas. The common information section 301 includes the video file name, the name of the camera used to capture the image, the name of the lens used during capture, the frame rate set during capture, and the image size of the video. The additional information section 302 records the offset frame number for the timed metadata (described below). The frame-by-frame information section 303 records the F-number, focal length, and lens metadata in association with the frame number of each frame. The lens metadata is the amount of correction to be applied to the distance from the center position of the image. In this embodiment, the frame-by-frame information section 303 records multiple pieces of lens metadata indicating the optimal amount of correction for each of the multiple frame images constituting the camera footage clip 202.

[0017] Upon completion of the lens aberration removal process, PC 100 chronologically combines CG material clip 201 and camera material clip 202 from which lens aberration has been removed to create combined clip 205 shown in Fig. 2(c). As shown in Fig. 2(c), combined clip 205 is made up of CG material clip section 205a corresponding to CG material clip 201 and camera material clip section 205b corresponding to camera material clip 202. Combined clip 205 is in a state where no lens aberration has occurred (for example, a state where there is no distortion aberration (see image 206 in Fig. 2(c))).

[0018] Next, the PC 100 performs range aberration application processing on the combined clip 205. At this time, if the PC 100 uses only the lens aberration metadata 300 as in the past, the lens aberration cannot be applied to the CG material clip section 205a because the lens aberration metadata 300 only includes the lens metadata of the camera material clip 202. In other words, in the combined clip 205, as shown in FIG. 2(d), lens aberration is applied only to the camera material clip section 205b. As a result, the CG material clip section 205a cannot be given an atmosphere of realism, and a sense of discontinuity occurs at the transition between the CG material clip section 205a and the camera material clip section 205b.

[0019] In contrast to this, in the first embodiment, a frame image to which lens aberration has not been applied is obtained from the combined clip 205, the frame number of the frame image is obtained, and a focal length value set in advance in association with this frame number is obtained from the virtual camera information 501. Furthermore, lens metadata corresponding to the focal length value closest to the above focal length value is obtained from the lens aberration metadata 300 of the camera material clip unit 205b, and lens aberration is applied to this frame image using this lens metadata.

[0020] FIG. 4 is a flowchart showing the steps of the lens aberration application process executed by the information processing device 100 according to the first embodiment. The lens aberration application process of FIG. 4 is implemented by the control unit 101 loading a program stored in the ROM 102 or the like into the RAM 103 and executing it. In this embodiment, as an example, the lens aberration application process is executed on the combined clip 205. As described above, the combined clip 205 is composed of a CG material clip section 205a and a camera material clip section 205b, and the camera material clip section 205b is the camera material clip 202 from which the lens aberration has been removed. Furthermore, as shown in FIG. 5(a), in the combined clip 205, the CG material clip section 205a is composed of 10 frames, from the 0th frame to the 9th frame of the combined clip 205. Furthermore, the camera material clip section 205b of the combined clip 205 is composed of 10 frames, from the 10th frame to the 19th frame of the combined clip 205. 4 is executed when the user operates the operation unit 105 to specify lens aberration metadata 300 corresponding to the combined clip 205. Note that the lens aberration metadata 300 is lens aberration metadata of the camera material clip 202 that constitutes the camera material clip portion 205b in the combined clip 205.

[0021] 4, first, the control unit 101 reads the lens aberration metadata 300 specified by the user (S401), and then the control unit 101 analyzes the read lens aberration metadata 300 (S402).

[0022] Next, the control unit 101 acquires virtual camera information 501 of FIG. 5(b) of the combined clip 205 (S403). The virtual camera information 501 is a table that records shooting setting values ​​for each frame in the combined clip 205, and is used to determine which lens metadata to use for each frame of the combined clip 205 when applying lens aberration. In this embodiment, before starting this process, the user operates the operation unit 105 to set shooting setting values ​​for each frame in the combined clip 205 to be recorded in the virtual camera information 501. The virtual camera information 501, including the shooting setting values ​​for each frame set by the user, is stored in the storage device 104, etc. In FIG. 5(b), focal length is recorded as an example of a shooting setting value, and the focal length is set to "50" for all frames. Note that in this embodiment, a configuration will be described in which the same value is set for the focal length of the virtual camera information 501 for all frames, but a different value may be set for each frame. However, if the difference in the focal length value in virtual camera information 501 between adjacent frames is very large, there is a concern that the degree of distortion and the amount of peripheral light falloff will differ significantly between these frames, resulting in an unnatural video. For this reason, when setting different values ​​for the focal length in virtual camera information 501 for each frame, it is necessary to set the focal length values ​​so that the difference between the focal lengths between adjacent frames does not become too large.

[0023] Next, the control unit 101 acquires, from the combined clip 205, the frame number to which lens aberration has not been applied and the frame image corresponding to this frame number (S404).

[0024] Next, the control unit 101 acquires the focal length corresponding to the frame number acquired in S404 from the virtual camera information 501 (S405). For example, if the frame number acquired in S404 is "0", the control unit 101 acquires "50", which is the focal length corresponding to the frame number "0", from the virtual camera information 501.

[0025] Next, the control unit 101 identifies the frame number corresponding to the focal length value closest to the focal length value acquired in S405 from the lens aberration metadata 300 (S406). For example, if the focal length value acquired in S405 is "50," the focal length value closest to this value in the frame unit information section 303 of the lens aberration metadata 300 in Fig. 3 is "50," and the frame number corresponding to this value is "1." In this way, if the focal length value acquired in S405 is "50," in S406 the control unit 101 identifies the frame number corresponding to the focal length value closest to the focal length value acquired in S405 as "1."

[0026] Next, the control unit 101 acquires the lens metadata corresponding to the frame number identified in S406 from the frame-by-frame information section 303 of the lens aberration metadata 300 (S407). For example, if the frame number identified in S406 is "1", the control unit 101 acquires the lens metadata corresponding to the frame number "1" from the frame-by-frame information section 303 of the lens aberration metadata 300.

[0027] Next, the control unit 101 applies lens aberration to the frame images acquired in S404 using the lens metadata acquired in S407 (S408). Next, the control unit 101 determines whether lens aberration has been applied to all frames of the combined clip 205 (S409).

[0028] If it is determined in S409 that lens aberration is not to be applied to any frame of the combined clip 205, the process returns to S404, and the frame numbers to which lens aberration has not been applied and the frame images corresponding to these frame numbers are acquired from the combined clip 205. In this manner, lens aberration is applied to all frames of the combined clip 205. In the present embodiment, as an example, the focal length value for all frames in the virtual camera information 501 is set to "50." Therefore, lens aberration is applied to each frame of the combined clip 205 using the lens metadata for frame number "1" corresponding to the focal length value closest to this focal length value (see, for example, FIG. 5(c)). As a result, in the combined clip 205, lens aberration is applied to both the CG material clip portion 205a and the camera material clip portion 205b, as shown in FIG. 6.

[0029] If it is determined in S409 that the lens aberration has been applied to all frames of the combined clip 205, this process ends.

[0030] According to the first embodiment described above, a frame image to which lens aberration has not been applied is obtained from the combined clip 205, the frame number of the frame image is obtained, and a preset focal length value associated with this frame number is obtained from the virtual camera information 501. Furthermore, lens metadata corresponding to the focal length value closest to the above-mentioned focal length value is obtained from the lens aberration metadata 300 of the camera material clip section 205b, and lens aberration is applied to this frame image using this lens metadata. That is, lens aberration is applied not only to the camera material clip section 205b but also to the CG material clip section 205a. This makes it possible to impart a realistic feel to the CG material clip section 205a in the combined clip 205. Furthermore, when applying lens aberration, the lens metadata included in the lens aberration metadata 300 of the camera material clip section 205b is used, and therefore lens aberration is applied to the CG material clip section 205a in the same manner as to the camera material clip section 205b. This reduces the sense of discontinuity at the transition between the CG material clip section 205a and the camera material clip section 205b. Therefore, in the first embodiment described above, in the combined clip 205, it is possible to reduce the sense of discontinuity at the transition between the CG material clip section 205a and the camera material clip section 205b while giving the CG material clip section 205a an atmosphere of real-life footage.

[0031] In the first embodiment described above, the virtual camera information 501 is set by the user, which allows the user's intention to be reflected in the degree of distortion and peripheral light falloff to be added to the combined clip 205.

[0032] In the first embodiment described above, a configuration has been described in which a focal length value is set as a shooting setting value in virtual camera information 501, but the configuration of virtual camera information 501 is not limited to this. For example, virtual camera information 501 may be configured to include values ​​other than focal length as shooting setting values. Examples of values ​​other than focal length include subject distance information such as an F-number or a T-number that affects the image captured by the camera, and shooting setting values ​​that affect the optical system.

[0033] In this embodiment, the virtual camera information 501 may also be configured to include multiple types of values, such as both the focal length value and the F-number, as shooting setting values.

[0034] Furthermore, in this embodiment, the user may be allowed to select items (focal length and F-number) to be included in the virtual camera information 501 as shooting setting values.

[0035] Furthermore, in this embodiment, the file structure of the lens aberration metadata 300 is not limited to the structure shown in FIG. 3 described above, and may be a structure such as JSON format.

[0036] The lens aberration metadata 300 may be recorded in a moving image file by a digital camera when capturing a moving image, or may be created by the information processing device 100 from a moving image file in which metadata for each frame is recorded.

[0037] In the embodiment described above, the information processing device 100 is configured to read a file in which the lens aberration metadata 300 is recorded, but the configuration is not limited to this. For example, the lens aberration metadata 300 may be embedded in the moving image file itself as timed metadata that changes for each frame, and the information processing device 100 may acquire the lens aberration metadata 300 from the moving image file.

[0038] In addition, in this embodiment, the user may be allowed to select the lens aberration to be applied, and the items (focal length and F-number) to be included as shooting setting values ​​in the virtual camera information 501 may be determined according to the lens aberration to be applied.

[0039] Next, an information processing device and a control method thereof according to a second embodiment of the present invention will be described.

[0040] The second embodiment is basically the same as the first embodiment in terms of configuration and operation, but differs from the first embodiment in that it applies lens aberration to all frame images that make up a combined clip without using virtual camera information. Therefore, a description of the overlapping configuration and operation will be omitted, and the following description will focus on the different configuration and operation.

[0041] FIG. 7 is a flowchart showing the steps of lens aberration application processing executed by the information processing device 100 in the second embodiment. The lens aberration application processing in FIG. 7 is similar to the lens aberration application processing in FIG. 4 described above, and the following will particularly describe the differences from the lens aberration application processing in FIG. 4 described above. Like the lens aberration application processing in FIG. 4 described above, the lens aberration application processing in FIG. 7 is also implemented by the control unit 101 loading a program stored in the ROM 102 or the like into the RAM 103 and executing it. In this embodiment, as an example, the lens aberration application processing is executed on the combined clip 801 in FIG. 8. The combined clip 801 is composed of a CG material clip section 801a, a camera material clip section 801b, and a CG material clip section 801c, and the camera material clip section 801b is the camera material clip 202 from which lens aberration has been removed. The CG material clip section 801a is composed of 10 frames from the 0th frame to the 9th frame of the combined clip 801. The camera material clip section 801b is made up of 10 frames, from the 10th frame to the 19th frame, of the combined clip 801. The CG material clip section 801c is made up of 10 frames, from the 20th frame to the 29th frame, of the combined clip 801. The lens aberration application process in Fig. 7 is executed when the user operates the operation unit 105 to specify lens aberration metadata 300 corresponding to the combined clip 801. Note that the lens aberration metadata 300 is lens aberration metadata of the camera material clip 202 that makes up the camera material clip section 801b in the combined clip 801.

[0042] 7, S701 and S702 are performed, which are the same processes as S401 and S402 described above. Next, the control unit 101 acquires the number of offset frames of the timed metadata from the additional information section 302 of the lens aberration metadata 300 (S703). This number of offset frames is a value that indicates the start frame position of the camera material clip section 801b in the combined clip 801. In this embodiment, since the CG material clip section 801a made up of 10 frames is placed before the first frame of the camera material clip section 801b, the additional information section 302 sets "10" as the number of offset frames of the timed metadata.

[0043] Next, the control unit 101 acquires the frame number to which lens aberration has not been applied and the frame image corresponding to this frame number from the combined clip 801 (S704). Next, the control unit 101 determines whether or not the frame number acquired in S704 has reached the camera material clip unit 801b (S705). Specifically, the control unit 101 determines whether or not the frame number acquired in S704 has reached "10," which is the offset frame number acquired in S703.

[0044] In S705, if the frame number acquired in S704 does not reach "10," which is the offset frame number acquired in S703, it is determined that the frame number acquired in S704 does not reach the camera material clip section 801b. In this case, the process proceeds to S706.

[0045] In S706, the control unit 101 acquires lens metadata corresponding to the first frame from the frame-by-frame information unit 303 of the lens aberration metadata 300. As described above, in this embodiment, if the frame number acquired in S704 is not the frame number corresponding to the camera material clip unit 801b but is a frame number earlier than the first frame number in the camera material clip unit 801b, lens metadata corresponding to the first frame in the camera material clip unit 801b is acquired from the lens aberration metadata 300. Next, the processing proceeds to S711, which will be described later.

[0046] In S705, if the frame number acquired in S704 reaches "10," which is the offset frame number acquired in S703, it is determined that the frame number acquired in S704 has reached the camera material clip section 801b. In this case, the process proceeds to S707.

[0047] In S707, the control unit 101 converts the frame number acquired in S704 into a frame number corresponding to the lens aberration metadata 300. Specifically, the control unit 101 subtracts the number of offset frames in the timed metadata acquired in S703 from the frame number acquired in S704. For example, the number of offset frames in the timed metadata acquired in S703, "10," is subtracted from the frame number acquired in S704, "10," to obtain "0." "0" corresponds to the first frame number of the camera material clip section 801b in the lens aberration metadata 300. In this way, in S707, the frame number acquired in S704 is converted into a frame number corresponding to the lens aberration metadata 300.

[0048] Next, the control unit 101 determines whether the converted frame number exceeds the end of the camera material clip unit 801b (S708). Specifically, the control unit 101 determines whether the converted frame number exceeds "9", which is the frame number of the final frame of the camera material clip unit 801b.

[0049] In S708, if the converted frame number does not exceed "9," which is the frame number of the final frame of the camera material clip section 801b, it is determined that the converted frame number does not exceed the end of the camera material clip section 801b. In this case, the process proceeds to S709.

[0050] In S709, the control unit 101 acquires lens metadata corresponding to the converted frame number from the frame unit information section 303 of the lens aberration metadata 300. As described above, in this embodiment, if the frame number acquired in S704 is the frame number corresponding to the camera material clip section 801b, lens metadata indicating the amount of lens aberration correction for the frame image acquired in S704 is acquired from the lens aberration metadata 300. Next, the processing proceeds to S711, which will be described later.

[0051] In S708, if the converted frame number exceeds "9," which is the frame number of the final frame in the camera material clip section 801b, it is determined that the converted frame number has exceeded the end of the camera material clip section 801b. In this case, the process proceeds to S710.

[0052] In S710, the control unit 101 acquires lens metadata corresponding to the final frame from the frame-by-frame information unit 303 of the lens aberration metadata 300. As described above, in this embodiment, if the frame number acquired in S704 is not the frame number corresponding to the camera material clip unit 801b but is a frame number subsequent to the final frame number in the camera material clip unit 801b, lens metadata corresponding to the final frame in the camera material clip unit 801b is acquired from the lens aberration metadata 300.

[0053] Next, the control unit 101 applies lens aberration to the frame image acquired in S704 using the lens metadata acquired in S706, S709, or S710.

[0054] For example, for the CG material clip section 801a arranged before the first frame of the camera material clip section 801b, lens aberration is applied using the lens metadata acquired in S706, i.e., the lens metadata of the first frame of the camera material clip section 801b (see, for example, 803 in FIG. 8(b)). Note that 802 in FIG. 8(b) indicates lens metadata corresponding to each frame number included in the frame-by-frame information section 303 of the lens aberration metadata 300.

[0055] Furthermore, lens aberration is applied to the camera material clip portion 801b using the lens metadata acquired in S709, that is, the lens metadata corresponding to each frame number of the camera material clip portion 801b (see, for example, 804 in FIG. 8(b)).

[0056] Furthermore, for the CG material clip section 801c, which is placed after the final frame of the camera material clip section 801b, lens aberration is applied using the lens metadata acquired in S710, i.e., the lens metadata of the final frame of the camera material clip section 801b (see, for example, 805 in FIG. 8(b)).

[0057] Next, the control unit 101 determines whether or not the lens aberration has been applied to all frames of the combined clip 801 (S712).

[0058] If it is determined in S712 that lens aberration is not to be applied to any frame of the combined clip 801, the process returns to S704, and the frame numbers to which lens aberration has not been applied and the frame images corresponding to these frame numbers are obtained from the combined clip 801. In this way, lens aberration is applied to all frames of the combined clip 801. As a result, in the combined clip 801, lens aberration is applied to all of the CG material clip section 801a, camera material clip section 801b, and CG material clip section 801c, as shown in FIG.

[0059] If it is determined in S712 that lens aberration has been applied to all frames of the combined clip 801, this process ends.

[0060] In the second embodiment described above, a frame image to which lens aberration has not been applied is obtained from the combined clip 801, and the frame number of that frame image is obtained. If this frame number is a frame number corresponding to the camera material clip section 801b, lens metadata indicating the amount of lens aberration correction for this frame image is obtained from the lens aberration metadata 300 of the camera material clip section 801b, and lens aberration is applied to this frame image using this lens metadata. Also, if this frame number is not a frame number corresponding to the camera material clip section 801b but is a frame number before the first frame number in the camera material clip section 801b, lens metadata indicating the amount of lens aberration correction for the first frame image in the camera material clip section 801b is obtained from the lens aberration metadata 300 of the camera material clip section 801b, and lens aberration is applied to this frame image using this lens metadata. Furthermore, if this frame number is not the frame number corresponding to the camera material clip section 801b but is a frame number subsequent to the final frame number in the camera material clip section 801b, lens metadata indicating the amount of lens aberration correction for the final frame image in the camera material clip section 801b is obtained from the lens aberration metadata 300 of the camera material clip section 801b, and lens aberration is applied to this frame image using this lens metadata. That is, lens aberration is applied not only to the camera material clip section 801b, but also to the CG material clip sections 801a and 801c. This allows the CG material clip sections 801a and 801c in the combined clip 801 to have a realistic feel. Furthermore, lens aberration is applied to the CG material clip sections 801a and 801c, respectively, using lens metadata indicating the amount of lens aberration correction for the frame image in the adjacent camera material clip section 205b. This reduces the sense of discontinuity at the transition to the camera material clip section 801b.Therefore, in the second embodiment described above, the CG material clip portion in the combined clip 801 can be given an atmosphere of real life, while reducing the sense of discontinuity at the transition between the CG material clip portion and the camera material clip portion.

[0061] Furthermore, in the second embodiment described above, it is determined whether or not the acquired frame number is the frame number corresponding to the camera material clip portion 801b by using the number of offset frames of the time domain for specifying the start frame of the camera material clip portion 801b in the combined clip 801. This makes it possible to reliably determine whether or not the acquired frame number is the frame number corresponding to the camera material clip portion 801b.

[0062] In the second embodiment described above, the lens aberration metadata 300 includes the number of offset frames of the timed metadata, but the present invention is not limited to this. For example, the number of offset frames of the timed metadata may be set by the user on a setting screen of an image editing application.

[0063] Next, an information processing device and a control method thereof according to a third embodiment of the present invention will be described.

[0064] The third embodiment is basically the same in configuration and operation as the first and second embodiments described above, but differs from the first and second embodiments in that the control for applying lens aberration is switched based on whether or not virtual camera information 501 has been acquired. Therefore, a description of the overlapping configuration and operation will be omitted, and the following description will focus on the different configuration and operation.

[0065] FIG. 10 is a flowchart showing the steps of lens aberration application processing executed by the information processing device 100 according to the third embodiment. The lens aberration application processing of FIG. 10 is similar to the lens aberration application processing of FIGS. 4 and 7 described above. Below, differences from the lens aberration application processing of FIGS. 4 and 7 will be described in particular. The lens aberration application processing of FIG. 10 is also implemented by the control unit 101 loading a program stored in the ROM 102 or the like onto the RAM 103 and executing it, similar to the lens aberration application processing of FIGS. 4 and 7 described above. In this embodiment, as an example, the lens aberration application processing is executed on the combined clip 205. The lens aberration application processing of FIG. 10 is executed when the user operates the operation unit 105 to specify lens aberration metadata 300 corresponding to the combined clip 205. The lens aberration metadata 300 is lens aberration metadata of the camera material clip 202 that constitutes the camera material clip section 205b in the combined clip 205.

[0066] 10, first, S1001 and S1002, which are the same processes as S401 and S402 described above, are performed. Next, the control unit 101 attempts to acquire the virtual camera information 501, and determines whether or not the virtual camera information 501 has been acquired (S1003).

[0067] If it is determined in S1003 that the virtual camera information 501 has been acquired, the control unit 101 performs a first control process. In the first control process, the above-described steps S404 to S409 are performed. Thereafter, this process ends.

[0068] If it is determined in S1003 that the virtual camera information 501 is not to be acquired, the control unit 101 performs the second control process. In the second control process, the above-described steps S703 to S712 are performed. Thereafter, this process ends.

[0069] In this way, even in a configuration in which control for applying lens aberration is switched based on whether or not the virtual camera information 501 has been acquired, the same effects as those of the above-described embodiment can be achieved.

[0070] In the present embodiment, the present invention has been described as being applied to a PC, which is an example of the information processing device 100, but the present invention can also be applied to a smartphone or a tablet terminal.

[0071] Although the present embodiment has been described with reference to an example in which a CG material clip composed of a CG image is used as a material clip composed of an image without lens aberration, the present invention is not limited to this. The present invention can also be applied to cases in which a material clip composed of an image other than a CG image that does not have lens aberration, such as an image read by a scanning device, is used. The present invention can also be applied to digital cameras that capture such material clips.

[0072] While the present invention has been described in detail above based on preferred embodiments, it is not limited to these specific embodiments, and various modifications within the spirit and scope of the present invention are also encompassed by the present invention. Parts of the above-described embodiments may be combined as appropriate. Furthermore, the present invention also encompasses a case in which a software program implementing the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program, either directly from a recording medium or via wired or wireless communication, and the program is then executed. Therefore, the program code itself supplied to and installed on a computer to implement the functional processing of the present invention also embodies the present invention. In other words, the computer program itself for implementing the functional processing of the present invention is also encompassed by the present invention. In this case, the program may take any form, such as object code, a program executed by an interpreter, or script data supplied to an OS, as long as it has the program functionality. Examples of recording media for providing the program include magnetic recording media such as hard disks and magnetic tapes, optical / magneto-optical storage media, and nonvolatile semiconductor memory. Another conceivable method for providing the program is to store the computer program implementing the present invention on a server on a computer network, and then download the computer program to a connected client computer.

[0073] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An information processing device that generates a combined clip by combining a first clip obtained by removing lens aberration from a video generated by shooting with a shooting device with a second clip composed of images without lens aberration, characterized in that the information processing device comprises: means for reading metadata of the first clip, in which multiple lens metadata corresponding to multiple frame images that constitute the first clip, each lens metadata indicating the amount of lens aberration correction, are recorded in association with the shooting setting values ​​used in the shooting; means for acquiring a frame image to which lens aberration has not been applied from the multiple frame images that constitute the combined clip, and acquiring the frame number of the frame image in the combined clip; means for acquiring a first shooting setting value that is preset in association with the acquired frame number; and means for acquiring lens metadata from the metadata of the first clip that corresponds to a second shooting setting value that is closest to the acquired first shooting setting value, and performing processing to apply lens aberration to the acquired frame image using the lens metadata. (Configuration 2) The information processing device according to configuration 1, wherein the first shooting setting value is set by a user. (Configuration 3) The information processing device according to configuration 1 or 2, wherein the first shooting setting value is at least one of a focal length value and an F-number. (Configuration 4) The apparatus further includes means for determining whether the first shooting setting values ​​have been acquired, and when it is determined that the first shooting setting values ​​have not been acquired, the means for performing the applying process applies lens aberration without using the first shooting setting values. In the process of applying lens aberration without using the first shooting setting values, when the acquired frame number is a frame number corresponding to the first clip, the means for performing the applying process acquires lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata. When the acquired frame number is not a frame number corresponding to the first clip, and a frame number earlier than the first frame number in the first clip is used, the means for performing the applying process acquires lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata. The information processing device of any one of configurations 1 to 3, characterized in that if the first clip is a frame number corresponding to the first clip, the means for performing the application process acquires lens metadata indicating the amount of lens aberration correction for the first frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform a process of applying lens aberration to the acquired frame image, and if the acquired frame number is not a frame number corresponding to the first clip but is a frame number subsequent to the final frame number in the first clip, the means for performing the application process acquires lens metadata indicating the amount of lens aberration correction for the final frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform a process of applying lens aberration to the acquired frame image. (Configuration 5) An information processing device that generates a combined clip by combining a first clip obtained by removing lens aberration from a moving image captured by a shooting device with a second clip made up of images without lens aberration, the information processing device comprising: means for reading metadata of the first clip in which a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, respectively, is recorded, the plurality of lens metadata indicating a correction amount for lens aberration; means for acquiring a frame image to which lens aberration has not been applied from the plurality of frame images constituting the combined clip and acquiring a frame number of the frame image in the combined clip; and means for applying lens aberration to the acquired frame image, wherein when the acquired frame number is a frame number corresponding to the first clip, the means for applying lens aberration acquires lens metadata indicating a correction amount for lens aberration for the acquired frame image from the metadata of the first clip, and uses the lens metadata to perform the processing of the acquired frame image. an information processing device that performs a process of applying lens aberration to a frame image, and if the acquired frame number is not a frame number corresponding to the first clip but is a frame number earlier than the first frame number in the first clip, the means for performing the applying process acquires lens metadata indicating the amount of lens aberration correction for the first frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform a process of applying lens aberration to the acquired frame image, and if the acquired frame number is not a frame number corresponding to the first clip but is a frame number later than the last frame number in the first clip, the means for performing the applying process acquires lens metadata indicating the amount of lens aberration correction for the last frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform a process of applying lens aberration to the acquired frame image. (Configuration 6) The information processing device according to Configuration 5, further comprising a means for determining whether the acquired frame number is a frame number corresponding to the first clip, wherein the determining means determines whether the acquired frame number is a frame number corresponding to the first clip using information identifying a start frame of the first clip in the combined clip. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the lens aberration is at least one of peripheral light falloff and distortion. (Configuration 8) The information processing device according to any one of configurations 1 to 7, wherein the image without lens aberration is a CG image. [Explanation of symbols]

[0074] 100 Information processing device 101 Control section 205, 801 Joint Clip 205a, 801a, 801c CG material clip section 205b, 801b camera material clip 300 Lens Aberration Metadata 501 Virtual Camera Information

Claims

1. An information processing device that generates a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image generated by shooting with a shooting device and a second clip made up of images without lens aberration, a means for reading the metadata of the first clip, in which a plurality of lens metadata items corresponding to a plurality of frame images constituting the first clip, respectively, and indicating correction amounts of lens aberrations are recorded in association with shooting setting values ​​used in the shooting; a means for acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; means for acquiring a first photographing setting value that is set in advance in association with the acquired frame number; and means for acquiring lens metadata corresponding to second shooting setting values ​​that are closest to the acquired first shooting setting values ​​from the metadata of the first clip, and performing processing to apply lens aberration to the acquired frame image using the lens metadata.

2. 2. The information processing apparatus according to claim 1, wherein the first photographing setting value is set by a user.

3. 2. The information processing apparatus according to claim 1, wherein the first photographing setting value is at least one of a focal length value and an F-number.

4. The apparatus further includes a means for determining whether the first photographing setting value has been acquired, When it is determined that the first photographing setting value is not to be acquired, the means for performing the application process performs a process of applying lens aberration without using the first photographing setting value, In the process of applying lens aberration without using the first shooting setting value, When the acquired frame number is a frame number corresponding to the first clip, the means for performing the applying process acquires lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata; If the acquired frame number is not a frame number corresponding to the first clip and is a frame number prior to the first frame number in the first clip, the means for performing the application process acquires lens metadata indicating a correction amount of lens aberration of the first frame image in the first clip from the metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata; The information processing device described in claim 1, characterized in that if the acquired frame number is not a frame number corresponding to the first clip but is a frame number later than the final frame number in the first clip, the means for performing the application processing acquires lens metadata indicating the amount of lens aberration correction for the final frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform processing to apply lens aberration to the acquired frame image.

5. An information processing device that generates a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image generated by shooting with a shooting device and a second clip made up of images without lens aberration, a means for reading the metadata of the first clip, which has recorded thereon a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, the lens metadata indicating a correction amount of lens aberration; a means for acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; a means for performing a process of applying lens aberration to the acquired frame image, When the acquired frame number is a frame number corresponding to the first clip, the means for performing the applying process acquires lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata; If the acquired frame number is not a frame number corresponding to the first clip and is a frame number prior to the first frame number in the first clip, the means for performing the application process acquires lens metadata indicating a correction amount of lens aberration of the first frame image in the first clip from the metadata of the first clip, and performs a process of applying lens aberration to the acquired frame image using the lens metadata; An information processing device characterized in that, if the acquired frame number is not a frame number corresponding to the first clip but is a frame number later than the final frame number in the first clip, the means for performing the application processing acquires lens metadata indicating the amount of lens aberration correction for the final frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform processing to apply lens aberration to the acquired frame image.

6. The method further includes determining whether the acquired frame number is a frame number corresponding to the first clip, 6. The information processing apparatus according to claim 5, wherein the determining means determines whether the acquired frame number is a frame number corresponding to the first clip by using information that identifies a start frame of the first clip in the combined clip.

7. 6. The information processing apparatus according to claim 1, wherein the lens aberration is at least one of peripheral light falloff and distortion.

8. 6. The information processing apparatus according to claim 1, wherein the image without lens aberration is a CG image.

9. A control method for an information processing device that generates a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image captured by an imaging device and a second clip made up of images without lens aberration, the method comprising: a step of reading the metadata of the first clip, in which a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, respectively, and indicating a correction amount of lens aberration, are recorded in association with shooting setting values ​​used in the shooting; acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; acquiring a first photographing setting value that is set in advance in association with the acquired frame number; and acquiring, from the metadata of the first clip, lens metadata corresponding to second shooting setting values ​​that are closest to the acquired first shooting setting values, and using the lens metadata to perform processing to apply lens aberration to the acquired frame image.

10. A control method for an information processing device that generates a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image captured by an imaging device and a second clip made up of images without lens aberration, the method comprising: reading the metadata of the first clip, in which a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, respectively, and indicating a correction amount of lens aberration, is recorded; acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; performing a process of applying lens aberration to the acquired frame images; When the acquired frame number is a frame number corresponding to the first clip, the step of performing the applying process includes acquiring lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performing a process of applying lens aberration to the acquired frame image using the lens metadata; When the acquired frame number is not a frame number corresponding to the first clip and is a frame number prior to the first frame number in the first clip, the step of performing the applying process includes acquiring lens metadata indicating a correction amount of lens aberration of the first frame image in the first clip from the metadata of the first clip, and performing a process of applying lens aberration to the acquired frame image using the lens metadata; A control method for an information processing device, characterized in that if the acquired frame number is not a frame number corresponding to the first clip but is a frame number later than the final frame number in the first clip, the process of performing the application process includes acquiring lens metadata indicating the amount of lens aberration correction for the final frame image in the first clip from the metadata of the first clip, and using the lens metadata to perform a process of applying lens aberration to the acquired frame image.

11. A program for causing a computer to execute a control method of an information processing device for generating a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image generated by shooting with a shooting device and a second clip made up of images without lens aberration, the program comprising: The control method for the information processing device includes: a step of reading the metadata of the first clip, in which a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, respectively, and indicating a correction amount of lens aberration, are recorded in association with shooting setting values ​​used in the shooting; acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; acquiring a first photographing setting value that is set in advance in association with the acquired frame number; and acquiring lens metadata corresponding to second shooting setting values ​​that are closest to the acquired first shooting setting values ​​from the metadata of the first clip, and using the lens metadata to perform processing to apply lens aberration to the acquired frame image.

12. A program for causing a computer to execute a control method of an information processing device for generating a combined clip by chronologically arranging and combining a first clip obtained by removing lens aberration from a moving image generated by shooting with a shooting device and a second clip made up of images without lens aberration, the program comprising: The control method for the information processing device includes: reading the metadata of the first clip, in which a plurality of lens metadata corresponding to a plurality of frame images constituting the first clip, respectively, and indicating a correction amount of lens aberration, is recorded; acquiring a frame image to which lens aberration has not been applied from among a plurality of frame images constituting the combined clip, and acquiring a frame number of the frame image in the combined clip; performing a process of applying lens aberration to the acquired frame images; When the acquired frame number is a frame number corresponding to the first clip, the step of performing the applying process includes acquiring lens metadata indicating a correction amount of lens aberration of the acquired frame image from metadata of the first clip, and performing a process of applying lens aberration to the acquired frame image using the lens metadata; When the acquired frame number is not a frame number corresponding to the first clip and is a frame number prior to the first frame number in the first clip, the step of performing the applying process includes acquiring lens metadata indicating a correction amount of lens aberration of the first frame image in the first clip from the metadata of the first clip, and performing a process of applying lens aberration to the acquired frame image using the lens metadata; A program characterized in that, when the acquired frame number is not a frame number corresponding to the first clip but is a frame number later than the final frame number in the first clip, the process of performing the application process acquires lens metadata indicating the amount of lens aberration correction for the final frame image in the first clip from the metadata of the first clip, and uses the lens metadata to perform a process of applying lens aberration to the acquired frame image.

Images