Imaging device, information processing device, control method, program and system

Abstract

This technology enables the application of necessary optical aberration corrections during image synthesis while suppressing image quality degradation. [Solution] The imaging device includes: an imaging means for capturing an image of a subject through an optical system; an image processing means capable of applying a plurality of correction processes relating to optical aberrations of the optical system to the image obtained by the imaging means; a recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image; a format for the image to be recorded in the recording means; and a control means for recording correction data in the recording means that allows correction processes not applied to the image to be applied to the image, based on correction processes not applied to the image.

Description

Technical Field

[0001] The present invention relates to a technique for synthesizing a real-shot image and CG.

Background Art

[0002] In virtual production and VFX (Visual Effects), when synthesizing an image captured by an imaging device such as a digital camera and computer graphics (CG), an operation is performed to remove optical aberrations (such as peripheral light quantity and distortion aberration) of the imaging optical system with respect to the captured image. Also, recently, due to the high degree of freedom in post-capture image processing, RAW videos are used as images to be synthesized with CG in virtual production and VFX.

[0003] Patent Document 1 describes an imaging device that records lens aberration correction data for correcting the aberration of a lens attached to a camera in association with a RAW video when shooting the RAW video. In Patent Document 1, when synthesizing a RAW video and CG, lens aberration correction data is acquired from the metadata attached to the RAW video before synthesis, and the synthesis is performed after removing the lens aberration.

[0004] In a digital camera, while it is possible to set correction of lens aberration in the development process of a RAW video, lens aberration correction is not applied in the digital camera, and correction data of lens aberration set in the digital camera is recorded as metadata in the RAW video file. By doing so, in the development process of the RAW video, optical aberration correction set in the digital camera can be applied.

[0005] In the development process of RAW video, pixel value corrections such as peripheral illumination correction and chromatic aberration correction (first correction process) are applied, followed by geometric transformations such as distortion correction (second correction process). In this case, the correction data used in the first correction process is only effective on the image before the second correction process is applied, and cannot be applied to the image after the second correction process has been applied. Furthermore, videos to be composited with CG must have both the first and second correction processes applied. For this reason, methods such as those described in Patent Document 2 have been proposed, in which an inverse transformation of distortion correction is applied to the image after distortion correction, and then chromatic aberration correction is applied. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2014-23063 [Patent Document 2] Japanese Patent Publication No. 2009-43060 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, Patent Document 2 suggests that applying the inverse transform of distortion correction may degrade image quality.

[0008] This invention has been made in view of the above problems, and its objective is to realize a technology that can apply necessary optical aberration correction while suppressing the degradation of image quality during image synthesis. [Means for solving the problem]

[0009] To solve the above problems and achieve the objective, the present invention provides an imaging apparatus comprising: imaging means for capturing an image of a subject through an optical system; image processing means capable of applying a plurality of correction processes relating to optical aberrations of the optical system to the image obtained by the imaging means; recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image; a format for the image to be recorded in the recording means; and control means for recording correction data in the recording means that allows correction processes not applied to the image to be applied to the image, based on correction processes not applied to the image. [Effects of the Invention]

[0010] According to the present invention, necessary optical aberration correction can be applied during image synthesis while suppressing degradation of image quality. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram illustrating the configuration of the system and apparatus of this embodiment. [Figure 2] A diagram illustrating the functions of the development application and image compositing application of this embodiment. [Figure 3] A flowchart illustrating the video recording process of this embodiment. [Figure 4] A flowchart illustrating the development process of this embodiment. [Figure 5] A flowchart illustrating the image synthesis process of this embodiment. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] In this embodiment, in a system in which an imaging device and an information processing device are communicably connected, an example will be described in which the imaging device captures a video, and the information processing device performs a process of synthesizing the video captured by the imaging device with a pre-created image such as computer graphics (CG) (hereinafter referred to as CG) in real time.

[0014] <Device Configuration> First, referring to FIG. 1(a), the configuration and functions of the imaging device according to this embodiment will be described.

[0015] The imaging device 100 according to this embodiment is, for example, a digital camera, a digital video camera, a smartphone or a tablet computer having a camera function and a communication function, or a web camera.

[0016] The functions of the imaging device 100 according to this embodiment are realized by the hardware shown in FIG. 1(a) and / or a software program executed by the control unit 101.

[0017] The imaging device 100 according to this embodiment includes a control unit 101, an imaging unit 102, an image processing unit 103, a non-volatile memory 104, a volatile memory 105, a storage 106, an operation unit 107, a display unit 108, and a communication I / F 109.

[0018] The control unit 101 is an arithmetic processing processor such as a CPU or an MPU that controls the entire imaging device 100, and realizes the control process described later by executing a program stored in the non-volatile memory 104 described later. Note that instead of the control unit 101 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0019] The imaging unit 102 has an imaging optical system. The imaging optical system includes a lens group including a zoom lens and a focus lens, and a shutter having an aperture function. Further, the imaging unit 102 includes an image sensor composed of a CCD, a CMOS element, etc. that converts a subject image into an electrical signal, and an A / D converter that converts an analog image signal output from the image sensor into a digital signal. The imaging unit 102, under the control of the control unit 101, converts the subject image light formed by the lenses included in the imaging unit 102 into an electrical signal by the image sensor, performs noise reduction processing, etc., and outputs a digital image signal. Note that the imaging optical system including the lens group and the shutter may be detachable from the imaging device 100.

[0020] The image processing unit 103 includes a GPU (Graphics Processing Unit) and executes various image processes including development processing on the RAW image signal of a still image or a moving image generated by the imaging unit 102. Further, the image processing unit 103 compresses and encodes the still image data subjected to image processing into JPEG or the like, or encodes the moving image data in a moving image compression format such as MP4 or MXF to create an image file, and records it in a storage 106 such as a memory card. Further, the image processing unit 103 records a RAW moving image file on which no image processing has been performed on the RAW image signal of a still image or a moving image generated by the imaging unit 102 in the storage 106. Further, the image processing unit 103 performs predetermined arithmetic processing using the image signal generated by the imaging unit 102, and based on the obtained arithmetic result, the control unit 101 controls the focus lens, aperture, and shutter of the imaging unit 102 to perform AF (auto focus) processing and AE (auto exposure) processing.

[0021] The image processing applicable to the image processing unit 103 includes pixel interpolation processing, color interpolation (demosaic) processing, white balance processing, and a plurality of types of correction processes related to optical aberrations of the imaging optical system. The correction process related to optical aberrations includes correction of pixel values (first correction process) such as peripheral light amount correction and / or magnification chromatic aberration correction, and geometric transformation (second correction process) such as distortion aberration correction and / or image magnification correction.

[0022] Furthermore, the correction data used for optical aberration correction includes first correction processing data used for the first correction processing, second correction data used for the second correction processing, and third correction data for applying the first correction processing to an image to which the second correction processing has been applied.

[0023] The imaging device 100 of this embodiment allows the user to individually set which optical aberration corrections to apply to the video being shot from among multiple optical aberration corrections, by pre-configuring them via a menu screen or the like. For example, when shooting video, it is possible to set it so that the first correction process is not applied, but the second correction process is applied (peripheral illumination correction off / distortion aberration correction on). Furthermore, when shooting video in an encoding format other than RAW, such as MP4 or MXF, the correction process that is set to "on" among the first and second correction processes will be applied.

[0024] The non-volatile memory 104 is a type of memory that allows data to be erased and written, such as flash memory, a solid-state drive (SSD), or a hard disk drive (HDD). Constants for the operation of the control unit 101, programs, etc., are stored in the non-volatile memory 104. The program referred to here is a program for executing the control processing described later in Figure 3.

[0025] The volatile memory 105 is RAM that temporarily stores programs read from the non-volatile memory 104, as well as constants and variables necessary for executing the programs. The volatile memory 105 is also used as working memory for the control unit 101 and the image processing unit 103. Furthermore, the volatile memory 105 is used as a buffer memory that temporarily holds image signals generated by the imaging unit 102 and image data processed by the image processing unit 103, and as image display memory for the display unit 108.

[0026] Storage 106 is a storage device such as a hard disk drive or solid state drive that is built into or can be connected to the imaging device 100. Alternatively, storage 106 can be implemented as an external storage device consisting of, for example, a media (recording medium) and a media drive that can access the media. The external storage device may also be a server device connected via a network. Storage 106 stores programs that implement the control processing described later in Figure 3, as well as image data such as still images and videos captured by the imaging device 100.

[0027] The operation unit 107 consists of various input devices such as switches, buttons, and dials that receive various operations from the user and output operation information to the control unit 101. The operation unit 107 includes, for example, a power button to turn the power on or off, a shooting button to start or stop taking still images or videos, a playback button to play back images, and a mode switching button to change the operating mode of the imaging device 100. The input devices may also be software buttons or keys using a touch display.

[0028] In still image capture mode, the camera performs AF (autofocus) processing and AE (automatic exposure) processing based on the image signal generated by the imaging unit 102. The control unit 101 also performs a shooting process to record the processed still image data as a still image file in the storage unit 106.

[0029] In video recording mode, the control unit 101 performs AF (autofocus) processing and AE (automatic exposure) processing based on the image signal for each frame generated by the imaging unit 102. The control unit 101 also performs shooting processing to record video data in a format other than RAW, for which image processing has been applied, as a video file, or RAW video data, for which no image processing has been applied, as a RAW video file to the storage 106. The control unit 101 starts the video recording process when the shooting button is pressed for the first time and continues the video recording process until the shooting button is pressed again. The control unit 101 stops the video recording process when the shooting button is pressed again and records the video data for the time from when the recording process started until when it was stopped to the storage 106.

[0030] The display unit 108 displays live view images generated by the imaging unit 102, captured images, images to be played back, and a GUI (Graphical User Interface) that accepts user operations. The display unit 108 is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 108 may be integrated with the imaging device 100 or it may be an external device connected to the imaging device 100. The imaging device 100 only needs to be able to connect to the display unit 108 and control the display of the display unit 108.

[0031] The communication interface (I / F) 109 communicates with the information processing device 200 via a wired or wireless method over a network such as the Internet or a LAN (Local Area Network). In this embodiment, the imaging device 100 transmits video files in formats other than RAW or RAW format stored in the storage 106 to the information processing device 200 via the communication I / F 109.

[0032] Next, with reference to Figure 1(b), the configuration and functions of the information processing device of this embodiment will be described.

[0033] The information processing device 200 in this embodiment is a general-purpose computer such as a personal computer (PC) or tablet computer, or a dedicated computer such as a server computer. The processing of the information processing device 200 in this embodiment may be implemented by a single computer device, or the functions may be distributed and implemented across multiple computer devices as needed. The multiple computer devices are connected to each other so as to be able to communicate with one another.

[0034] The functions of the information processing device 200 in this embodiment are realized by the hardware shown in Figure 1(b) and / or the software program executed by the control unit 201.

[0035] The information processing device 200 of this embodiment includes a control unit 201, a non-volatile memory 202, a volatile memory 203, a storage 204, an operation unit 205, a display unit 206, a communication interface 207, and a system bus 208.

[0036] The control unit 201 includes a CPU, MPU, or other arithmetic processing processor that comprehensively controls the entire information processing device 200. The non-volatile memory 202 is a program memory such as ROM that stores programs and parameters executed by the processor of the control unit 201. Here, the program refers to a program that realizes the functions of the application described later in Figure 2 and the control processing described later in Figures 4-5.

[0037] The volatile memory 203 is a data memory such as RAM that temporarily stores programs read from the non-volatile memory 202, as well as constants and variables necessary for executing the program. The volatile memory 203 includes the program work area of ​​the control unit 201, a backup area for error handling, and a program load area.

[0038] In this embodiment, program memory may be realized by loading a program from an external storage device connected to the information processing device 200 into the volatile memory 203.

[0039] Storage 204 is a storage device such as a hard disk drive or solid state drive that is built into or connectable to the information processing device 200. Alternatively, storage 204 can be implemented as an external storage device consisting of, for example, media (recording medium) and a media drive capable of accessing that media. The external storage device may also be a server device connected via a network. Storage 204 stores programs that implement the functions of the application described later in Figure 2 and the control processes described later in Figures 4-5, as well as image data such as still images and videos acquired from the imaging device 100 and external storage devices.

[0040] The operation unit 205 is an input device such as a mouse, touch panel, joystick, or keyboard that accepts user input and outputs operation instructions to the control unit 201.

[0041] The display unit 206 is an output device such as a display made of LCD or organic EL, and displays the application's GUI (Graphical User Interface), menu screen, captured image, various operation buttons, etc.

[0042] The communication interface (I / F) 207 communicates with the imaging device 100 via a network such as the Internet or a LAN (Local Area Network) using a wired or wireless method. In this embodiment, the information processing device 200 acquires video files in formats other than RAW or RAW format from the imaging device 100 or an external storage device via the communication I / F 207, and performs development processing and image synthesis processing.

[0043] The system bus 208 includes an address bus, a data bus, and a control bus that connect the components 201 to 207 of the information processing device 200 in a manner that enables the exchange of data.

[0044] The non-volatile memory 202 stores the operating system (OS), which is the basic software executed by the control unit 201, and programs including applications that work in cooperation with the OS to realize advanced functions.

[0045] The image development and image synthesis functions of the information processing device 200 in this embodiment are implemented by software provided by an application, which will be described later in Figure 2. The application is assumed to have software for utilizing the basic functions of the OS installed on the information processing device 200. The OS of the information processing device 200 may also have software for implementing the processing in this embodiment.

[0046] <Image development and image compositing processes> Next, with reference to Figure 2, the functions of the development application and image compositing application of this embodiment will be described.

[0047] Figure 2 illustrates the functions of the development application and image compositing application in this embodiment.

[0048] This embodiment describes an example where peripheral light intensity and distortion occur as optical aberrations in the imaging optical system of the imaging device 100, and where peripheral light intensity correction is set to off and distortion correction is set to on as the optical aberration correction set in the imaging device 100. In this embodiment, when a RAW video file is input to the information processing device 200, the development application 250 applies the distortion correction set to on in the imaging device 100 to perform development processing on the RAW video data.

[0049] It should be noted that the development process performed by the development application and the image compositing process performed by the image compositing application are independent and the development process is not necessarily based on the premise of image compositing. Therefore, even if the RAW video file has first, second, and third correction data added to it, the development process does not necessarily apply both the first and second correction processes, regardless of the optical aberration correction set in the imaging device 100. For example, in the development process, there are cases where image compositing is not anticipated, and only distortion aberration correction is applied without peripheral illumination correction, and CG compositing is performed at the editing stage. In that case, it is troublesome to apply the first and second correction processes again in the development process, so it is desirable that peripheral illumination correction can be applied in the image compositing process as in this embodiment. Also, image compositing is often performed by a different organization or team than the development process, and in that case, it is conceivable that image compositing may be performed using a developed video to which only distortion correction has been applied.

[0050] In Figure 2, the development application 250 receives a RAW video file from the imaging device 100 with distortion correction enabled in S251. In S252, it applies distortion correction using the second correction data attached to the input RAW video file and performs development processing, and in S253, it outputs the developed video with distortion correction applied.

[0051] The image compositing application 260 receives, in S261, a video file developed by the development application 250 or a video file shot in a format other than RAW with distortion correction enabled from the imaging device 100. In S263, a CG image is input, and in S264, the image compositing process of the video and CG is performed. Here, since CG does not have optical aberrations, it is necessary to remove optical aberrations from the video before image compositing. Since distortion correction has already been applied to the input video, peripheral illumination correction is applied using a third correction data in S262, and a compositable video with distortion correction and peripheral illumination correction applied is generated in S264. In S265, a video is generated by compositing the video with distortion correction and peripheral illumination correction applied and the CG image.

[0052] To revert a video created using image synthesis in S265 back to a video without peripheral illumination correction, apply the inverse transformation of peripheral illumination correction in S266. This will generate a video in S267 that combines the video with distortion correction applied and the CG.

[0053] In contrast, conventionally, it is common practice to apply peripheral illumination correction to a video in an image compositing application, and then apply distortion correction. In this case, the correction data used for peripheral illumination correction is only valid for the video before distortion correction is applied, and cannot be applied to the video after distortion correction has been applied. Therefore, peripheral illumination correction S261 in Figure 2 cannot be applied.

[0054] In this embodiment, the video input to the information processing device 200 is provided with correction data that allows peripheral illumination correction to be applied to a video to which distortion aberration correction has been applied. Therefore, in S261, peripheral illumination correction can be applied to a video to which distortion aberration correction has been applied.

[0055] <Video Recording Processing> Next, with reference to Figure 3, the video recording process using the imaging device 100 of this embodiment will be described.

[0056] Figure 3 illustrates a flowchart of the video recording process using the imaging device of this embodiment.

[0057] The process shown in Figure 3 is achieved when the control unit 101 loads a program stored in the non-volatile memory 104 into the volatile memory 105 and executes it, thereby controlling each component of the imaging device 100.

[0058] In this embodiment, we will describe an example in which peripheral light intensity correction (first correction process) and distortion aberration correction (second correction process) can be applied as optical aberration correction of the imaging optical system when the imaging device 100 is shooting video.

[0059] In step S300, the control unit 101 sets the frame count N (where N is an integer) to 1.

[0060] In step S301, the control unit 101 determines whether or not ambient light correction is set to ON. If it determines that it is set to ON, it proceeds to step S305; if it determines that it is not set to ON, it proceeds to step S310.

[0061] In step S305, the control unit 101 determines whether the video to be recorded is in RAW format. If it determines that it is in RAW format, it proceeds to step S310; otherwise, it proceeds to step S315.

[0062] In step S310, the control unit 101 records correction data for applying peripheral illumination correction to the Nth frame image as metadata for the captured video. The metadata related to peripheral illumination correction recorded here is correction data (first correction data) for applying peripheral illumination correction to the video before distortion aberration correction is applied. The first correction data is recorded regardless of whether the format of the video being shot is RAW or not, if the peripheral illumination correction setting is off. Also, if the peripheral illumination correction setting is on, the first correction data is recorded if the format of the video being shot is RAW, and not recorded if it is in a format other than RAW.

[0063] In step S315, the control unit 101 applies peripheral light correction to the Nth frame image.

[0064] In step S320, the control unit 101 determines whether or not distortion correction is set to ON. If it determines that it is set to ON, it proceeds to step S325; if it determines that it is not set to ON, it proceeds to step S340.

[0065] In step S325, the control unit 101 determines whether the video to be recorded is in RAW format. If it determines that it is in RAW format, it proceeds to step S330; otherwise, it proceeds to step S335.

[0066] In step S340, the control unit 101 determines whether the video to be recorded is in RAW format. If it determines that it is in RAW format, it proceeds to step S330; otherwise, it proceeds to step S345.

[0067] In step S345, the control unit 101 records correction data for applying distortion correction to the Nth frame image as metadata for the captured video. The metadata related to distortion correction recorded here is correction data (second correction data) for applying distortion correction to the video before and after peripheral illumination correction is applied. The second correction data is recorded regardless of whether the video being captured is in RAW format or a format other than RAW, if the distortion correction setting is turned off.

[0068] In step S330, the control unit 101 records correction data for applying distortion correction to the Nth frame image as metadata for the captured video. The metadata related to distortion correction recorded here is correction data (second correction data) for applying distortion correction to the video before and after peripheral illumination correction is applied. The second correction data is recorded when the distortion correction setting is on and the format of the video being shot is RAW, but not when the format is other than RAW.

[0069] In step S335, the control unit 101 applies distortion correction to the Nth frame image.

[0070] In step S350, the control unit 101 records correction data for applying peripheral illumination correction to the Nth frame image as metadata for the captured video. The metadata for peripheral illumination correction recorded here is correction data (third correction data) for applying peripheral illumination correction to the video after distortion correction has been applied. The third correction data is recorded when the format of the video being shot is RAW, regardless of whether the distortion correction setting is on or off. The third correction data is also recorded in the video to which distortion correction has been applied when the distortion correction setting is on and the format of the video being shot is a format other than RAW.

[0071] In step S355, the control unit 101 records the Nth frame image into a video file.

[0072] In step S360, the control unit 101 determines whether or not to end the shooting. If it determines to end the shooting, it terminates the process. If it determines not to end the shooting, it proceeds to step S365, increments the frame count N by 1, and returns to step S301.

[0073] The video file captured by the process shown in Figure 3, and the metadata recorded in the video file, are transmitted to the information processing device 200.

[0074] As shown in Figure 3, when the imaging device 100 of this embodiment captures RAW video, it does not perform development processing or apply optical aberration correction to the RAW video, but instead records the first correction data, second correction data, and third correction data used for optical aberration correction as metadata in the RAW video. In this way, optical aberration correction can be applied in the development processing and image synthesis processing of the RAW video by the information processing device 200.

[0075] Furthermore, the imaging device 100 of this embodiment records the video to be captured and correction data that can be applied to the video, based on the format of the video to be captured and the optical aberration correction that has not been applied to the video to be captured.

[0076] In more detail, when the imaging device 100 of this embodiment shoots video in a format other than RAW, it applies the optical aberration correction set to ON in the imaging device 100 and records the developed video data as a video file. Furthermore, the imaging device 100 of this embodiment records the correction data used for the unapplied optical aberration correction among the first correction data, second correction data, and third correction data as metadata in the video file.

[0077] In detail, when shooting video in a format other than RAW, if the first and second correction processes have not been applied, at least the first and second correction data are recorded as metadata. If the first correction process has been applied, at least the second correction data is recorded as metadata, and if the second correction process has been applied, at least the third correction data is recorded as metadata. If both the first and second correction processes have been applied, the first correction data, the second corrected data, and the third correction data are not recorded. In this way, optical aberration correction that has not been applied in the imaging device 100 can be applied in the image synthesis process of the information processing device 200.

[0078] <Development process> Next, with reference to Figure 4, the development process performed by the information processing device 200 of this embodiment will be described.

[0079] Figure 4 illustrates a flowchart of the development process performed by the information processing device 200 of this embodiment.

[0080] The process shown in Figure 4 is realized by the control unit 201 executing a development application and controlling each component of the information processing device 200.

[0081] Furthermore, in the process shown in Figure 4, the information processing device 200 acquires the video file captured by the process shown in Figure 3 and the metadata recorded in the video file from the imaging device 100 or an external storage device.

[0082] In this embodiment, an example is described in which the development application of the information processing device 200 can apply peripheral light intensity correction (first correction process) and distortion aberration correction (second correction process) in the development process of the input video.

[0083] In step S400, the control unit 201 determines whether or not a video file has been loaded. If the control unit 201 determines that a video file has been loaded, it proceeds to step S405; if it determines that a video file has not been loaded, it returns to step S400.

[0084] In step S405, the control unit 201 determines whether the video file read in step S400 is in RAW format. If the control unit 201 determines that it is in RAW format, it proceeds to step S410; otherwise, it terminates the process.

[0085] In step S410, the control unit 201 sets the frame count N (where N is an integer) to 1.

[0086] In step S415, the control unit 201 obtains the Nth frame image from the video file read in step S400.

[0087] In step S420, the control unit 201 determines whether the video file read in step S400 was shot with the peripheral illumination correction setting turned on. If the control unit 201 determines that the video file was shot with the peripheral illumination correction setting turned on, it proceeds to step S425. If it determines that the video file was not shot with the peripheral illumination correction setting turned on, it proceeds to step S435.

[0088] In step S425, the control unit 201 obtains correction data (first correction data) for applying peripheral illumination correction to the Nth frame image from the metadata of the video file read in step S400. The correction data obtained here for peripheral illumination correction is correction data (first correction data) for applying peripheral illumination correction to the image before distortion aberration correction is applied.

[0089] In step S430, the control unit 201 applies peripheral light correction to the Nth frame image using the correction data acquired in step S425.

[0090] In step S435, the control unit 201 determines whether the video file read in step S400 was shot with distortion correction enabled. If the control unit 201 determines that the video file was shot with distortion correction enabled, it proceeds to step S440; otherwise, it proceeds to step S450.

[0091] In step S440, the control unit 201 obtains correction data (second correction data) from the metadata of the video file read in step S400 to apply distortion correction to the Nth frame image.

[0092] In step S445, the control unit 201 applies distortion correction to the Nth frame image using the correction data acquired in step S440.

[0093] In step S450, the control unit 201 applies other development processing to the Nth frame image.

[0094] In step S455, the control unit 201 determines whether the Nth frame being processed is the final frame. If the control unit 201 determines that it is the final frame, it terminates the process. If it determines that it is not the final frame, it proceeds to step S460, increments the frame count N by 1, and returns to step S415.

[0095] The video data developed through the process shown in Figure 4 can be used for image compositing in image compositing applications.

[0096] <Image synthesis processing> Next, with reference to Figure 5, the image synthesis process by the information processing device 200 of this embodiment will be described.

[0097] Figure 5 illustrates a flowchart of the image synthesis process performed by the information processing device 200 of this embodiment.

[0098] The process shown in Figure 5 is realized by the control unit 201 executing an image synthesis application and controlling each component of the information processing device 200.

[0099] In this embodiment, an example is described in which the image synthesis application of the information processing device 200 can apply peripheral light intensity correction (first correction process) and distortion aberration correction (second correction process) to the developed video.

[0100] In step S500, the control unit 201 determines whether or not a video file has been loaded. If the control unit 201 determines that a video file has been loaded, it proceeds to step S505; if it determines that a video file has not been loaded, it returns to step S500. The video files loaded here are video files shot in a format other than RAW using the process shown in Figure 3, or video files that have been developed using the process shown in Figure 4.

[0101] In step S505, the control unit 201 sets the frame count N to 1.

[0102] In step S510, the control unit 201 obtains the Nth frame image from the video file read in step S500.

[0103] In step S515, the control unit 201 determines whether the Nth frame image acquired in step S510 is a frame to be processed for image synthesis. If the control unit 201 determines that it is a frame to be processed for image synthesis, it proceeds to step S525. If it determines that it is not a frame to be processed for image synthesis, it proceeds to step S520, increments the frame count N by 1, and returns to step S510.

[0104] In step S525, the control unit 201 determines whether or not peripheral illumination correction has been applied to the Nth frame image acquired in step S510. If the control unit 201 determines that peripheral illumination correction has been applied, it proceeds to step S530; if it determines that peripheral illumination correction has not been applied, it proceeds to step S545.

[0105] In step S530, the control unit 201 determines whether or not distortion correction has been applied to the Nth frame image acquired in step S510. If the control unit 201 determines that distortion correction has been applied, it proceeds to step S580; if it determines that distortion correction has not been applied, it proceeds to step S535.

[0106] In step S535, the control unit 201 obtains correction data (second correction data) from the metadata of the video file read in step S500 to apply distortion correction to the Nth frame image.

[0107] In step S540, the control unit 201 applies distortion correction to the Nth frame image using the correction data acquired in step S535.

[0108] In step S545, the control unit 201 determines whether or not distortion correction has been applied to the Nth frame image acquired in step S510. If the control unit 201 determines that distortion correction has been applied, it proceeds to step S550; if it determines that distortion correction has not been applied, it proceeds to step S560.

[0109] In step S550, the control unit 201 obtains correction data (third correction data) from the metadata of the video file read in step S500 to apply peripheral illumination correction to the image of the Nth frame to which distortion aberration correction has been applied.

[0110] In step S555, the control unit 201 applies peripheral illumination correction to the Nth frame image using the correction data acquired in step S550.

[0111] In step S560, the control unit 201 obtains correction data (first correction data) from the metadata of the video file read in step S500 to apply peripheral illumination correction to the image of the Nth frame before distortion aberration correction is applied.

[0112] In step S565, the control unit 201 applies peripheral illumination correction to the Nth frame image using the correction data acquired in step S560.

[0113] In step S570, the control unit 201 obtains correction data (second correction data) from the metadata of the video file read in step S500 to apply distortion correction to the Nth frame image.

[0114] In step S575, the control unit 201 applies distortion correction to the Nth frame image using the correction data acquired in step S570.

[0115] In step S580, the control unit 201 generates a composite image by combining the Nth frame image with the CG.

[0116] In step S585, the control unit 201 determines whether the Nth frame being processed is the final frame. If the control unit 201 determines that it is the final frame, it terminates the process. If it determines that it is not the final frame, it proceeds to step S520, increments the frame count N by 1, and returns to step S510.

[0117] Through the processes shown in Figures 4 and 5, the information processing device 200 of this embodiment applies any unapplied correction processing to the video to be composited with CG in the image compositing process, using applicable correction data.

[0118] More specifically, in this embodiment, the information processing device 200 performs development processing and image synthesis processing if the input video file is in RAW format. Furthermore, in this embodiment, the information processing device 200 does not perform development processing, but instead performs image synthesis processing if the input video file is in a format other than RAW.

[0119] Furthermore, if the input video file is in RAW format, the information processing device 200 of this embodiment performs development processing of the RAW video data, applies optical aberration correction using correction data recorded as metadata in the RAW video data, and synthesizes it with CG.

[0120] Furthermore, in this embodiment, if the input video file is in a format other than RAW, the information processing device 200 applies the first correction process using the third correction data if the second correction process has already been applied to the video data but the first correction process has not been applied during the image synthesis process. Also, if neither the first nor the second correction process has been applied to the video data, the first correction process is applied using the first correction data, and the second correction process is executed using the second correction data. In this way, the information processing device 200 of this embodiment can apply the first and second correction processes to the video data and then perform image synthesis with CG during the image synthesis process. Moreover, in this embodiment, even if the first correction process has not been applied to the video data but the second correction process has been applied, the information processing device 200 can apply the first correction process using the third correction data, and then perform image synthesis with CG after applying the first and second correction processes.

[0121] According to this embodiment, the imaging device 100 records the video to be captured and correction data applicable to the video, based on the format of the video to be captured and any optical aberration corrections not applied to the video. This allows the information processing device 200 to apply necessary optical aberration corrections while suppressing image quality degradation during image synthesis.

[0122] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0123] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0124] The disclosures herein include the following imaging devices, information processing devices, control methods, programs, and systems. [Item 1] An imaging device, An imaging means for capturing an image of a subject through an optical system, Image processing means capable of applying a plurality of correction processes relating to the optical aberrations of the optical system to the image obtained by the imaging means, A recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image, An imaging apparatus characterized by having a format for an image to be recorded in the recording means, and a control means for recording correction data in the recording means that allows correction processing not applied to the image to be applied to the image, based on correction processing not applied to the image. [Item 2] The imaging apparatus according to item 1, characterized in that the plurality of correction processes include a first correction process relating to the pixel values ​​of the image and a second correction process relating to the geometric transformation of the image. [Item 3] The imaging apparatus according to item 2, characterized in that the first correction process includes at least one of peripheral illumination correction and magnification chromatic aberration correction, and the second correction process includes at least one of distortion correction and image magnification correction. [Item 4] The imaging apparatus according to item 2 or 3, characterized in that, when recording an image in RAW format, the control means records correction data relating to the first correction process applied to the image before the second correction process is applied, correction data relating to the first correction process applied to the image after the second correction process has been applied, and correction data relating to the second correction process in the recording means. [Item 5] The imaging apparatus according to item 2 or 3, characterized in that, when the control means records an image of a first format other than RAW format, and the first correction process and the second correction process have not been applied to the image of the first format, the recording means records correction data relating to the first correction process applied to the image before the second correction process is applied, and correction data relating to the second correction process. [Item 6] The imaging apparatus according to item 2 or 3, characterized in that, when the control means records an image of a first format other than RAW format, and the first correction process is applied to the image of the first format, the control means records correction data relating to the second correction process in the recording means. [Item 7] The imaging apparatus according to item 2 or 3, characterized in that, when the control means records an image of a first format other than RAW format, and the second correction process is applied to the image of the first format, the control means records correction data relating to the first correction process that is applicable to the image to which the second correction process has been applied in the recording means. [Item 8] The imaging apparatus according to item 2 or 3, characterized in that when the control means records an image of a first format other than RAW format, and the first correction process and the second correction process are applied to the image of the first format, the recording means does not record the correction data related to the first correction process and the correction data related to the second correction process. [Item 9] The aforementioned image is a video, The imaging device according to any one of items 1 to 8, further comprising setting means for setting whether or not to apply each of the multiple correction processes to the frames of the video. [Item 10] An information processing device, A means for acquiring a first image, The system includes a synthesis means that applies at least one of a plurality of correction processes related to optical aberrations to the first image, and synthesizes a pre-created second image with the first image to which the plurality of correction processes have been applied, The information processing apparatus is characterized in that, when combining the first image and the second image, the combining means applies to the first image any unapplied correction processes among the plurality of correction processes that have not been applied to the first image, using correction data applicable to the first image. [Item 11] The information processing apparatus according to item 10, characterized in that the plurality of correction processes include a first correction process relating to the pixel values ​​of the image and a second correction process relating to the geometric transformation of the image. [Item 12] The information processing apparatus according to item 11, characterized in that the first correction process includes at least one of peripheral light intensity correction and magnification chromatic aberration correction, and the second correction process includes at least one of distortion correction and image magnification correction. [Item 13] The information processing device according to item 11 or 12, characterized by having a determination means for determining which of the plurality of correction processes has not been applied to the first image. [Item 14] The information processing apparatus according to item 13, characterized in that the synthesis means applies the first correction process if the first correction process has not been applied to the first image, and applies the second correction process if the second correction process has not been applied to the first image. [Item 15] The information processing apparatus according to item 13, characterized in that the synthesis means applies the second correction process to the first image, and if the first correction process has not been applied, applies the first correction process using correction data relating to the first correction process that can be applied to an image to which the second correction process has been applied. [Item 16] The information processing apparatus according to item 13, characterized in that, if the first correction process and the second correction process have not been applied to the first image, the synthesis means applies the first correction process to the first image using correction data related to the first correction process, and applies the second correction process to the first image using correction data related to the second correction process. [Item 17] The information processing apparatus according to any one of items 11 to 16, characterized in that the synthesis means generates a composite image to which only the second correction process has been applied by applying a process in the opposite direction to the first correction process applied to the first image to the composite image obtained by synthesizing the first image to which the plurality of correction processes have been applied and the second image. [Item 18] If the first image is a RAW video, the system has a development means that applies at least one of the plurality of correction processes to the RAW video to perform development processing. The information processing apparatus according to any one of items 10 to 17, characterized in that the synthesis means synthesizes the frames of the video that have undergone the development process with the second image. [Item 19] The information processing device according to any one of items 10 to 18, characterized in that the second image is computer graphics. [Item 20] A method for controlling an imaging device, The imaging device is An imaging means for capturing an image of a subject through an optical system, Image processing means capable of applying a plurality of correction processes relating to the optical aberrations of the optical system to the image obtained by the imaging means, The system includes recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image, The control method described above is A control method characterized by having the step of recording in the recording means correction data in which correction processing not applied to the image can be applied to the image, based on the format of the image to be recorded in the recording means and correction processing not applied to the image. [Item 21] A method for controlling an information processing device, The first step is to obtain the first image, The method includes the steps of applying at least one of a plurality of correction processes for optical aberrations to the first image, and compositing a pre-created second image onto the first image to which the plurality of correction processes have been applied, The control method is characterized in that, in the step of combining the first image and the second image, the unapplied correction processing among the plurality of correction processing that has not been applied to the first image is applied to the first image using correction data that can be applied to the first image. [Item 22] A program for causing a computer to function as one of the means of an imaging apparatus described in any of items 1 through 9. [Item 23] A program for causing a computer to function as one of the means of an information processing device described in any of items 10 to 19. [Item 24] An imaging device as described in any of items 1 to 9, A system including an information processing device as described in any of items 10 to 19. [Explanation of symbols]

[0125] 100...Imaging device, 101...Control unit, 102...Imaging unit, 103...Image processing unit, 200...Information processing device, 201...Control unit

Claims

1. An imaging device, An imaging means for capturing an image of a subject through an optical system, Image processing means capable of applying a plurality of correction processes relating to the optical aberrations of the optical system to the image obtained by the imaging means, A recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image, An imaging apparatus characterized by having a format for an image to be recorded in the recording means, and a control means for recording correction data in the recording means that allows correction processing not applied to the image to be applied to the image, based on correction processing not applied to the image.

2. The imaging apparatus according to claim 1, characterized in that the plurality of correction processes include a first correction process relating to the pixel values ​​of the image and a second correction process relating to the geometric transformation of the image.

3. The imaging apparatus according to claim 2, characterized in that the first correction process includes at least one of peripheral light intensity correction and magnification chromatic aberration correction, and the second correction process includes at least one of distortion correction and image magnification correction.

4. The imaging apparatus according to claim 2, characterized in that, when the control means records an image in RAW format, it records correction data relating to the first correction process applied to the image before the second correction process is applied, correction data relating to the first correction process applied to the image after the second correction process has been applied, and correction data relating to the second correction process in the recording means.

5. The imaging apparatus according to claim 2, characterized in that, when the control means records an image of a first format other than RAW format, and the first correction process and the second correction process have not been applied to the image of the first format, the recording means records correction data relating to the first correction process applied to the image before the second correction process is applied, and correction data relating to the second correction process.

6. The imaging apparatus according to claim 2, characterized in that, when the control means records an image of a first format other than RAW format, and the first correction process is applied to the image of the first format, the control means records correction data relating to the second correction process in the recording means.

7. The imaging apparatus according to claim 2, characterized in that, when the control means records an image of a first format other than RAW format, and the second correction process is applied to the image of the first format, the control means records correction data relating to the first correction process that is applicable to the image to which the second correction process is applied in the recording means.

8. The imaging apparatus according to claim 2, characterized in that when the control means records an image of a first format other than RAW format, and the first correction process and the second correction process are applied to the image of the first format, the correction data related to the first correction process and the correction data related to the second correction process are not recorded in the recording means.

9. The aforementioned image is a video, The imaging apparatus according to claim 1, further comprising setting means for setting whether or not to apply each of the plurality of correction processes to the frames of the video.

10. An information processing device, A means for acquiring a first image, The system includes a synthesis means that applies at least one of a plurality of correction processes related to optical aberrations to the first image, and synthesizes a pre-created second image with the first image to which the plurality of correction processes have been applied, The information processing apparatus is characterized in that, when combining the first image and the second image, the synthesis means applies to the first image any unapplied correction processes among the plurality of correction processes that have not been applied to the first image, using correction data applicable to the first image.

11. The information processing apparatus according to claim 10, characterized in that the plurality of correction processes include a first correction process relating to the pixel values ​​of the image and a second correction process relating to the geometric transformation of the image.

12. The information processing apparatus according to claim 11, characterized in that the first correction process includes at least one of peripheral light intensity correction and magnification chromatic aberration correction, and the second correction process includes at least one of distortion correction and image magnification correction.

13. The information processing apparatus according to claim 11, further comprising determination means for determining which of the plurality of correction processes has not been applied to the first image.

14. The information processing apparatus according to claim 13, characterized in that the synthesis means applies the first correction process if the first correction process has not been applied to the first image, and applies the second correction process if the second correction process has not been applied to the first image.

15. The information processing apparatus according to claim 13, characterized in that the synthesis means applies the second correction process to the first image, and if the first correction process has not been applied, applies the first correction process using correction data relating to the first correction process that can be applied to an image to which the second correction process has been applied.

16. The information processing apparatus according to claim 13, characterized in that the synthesis means applies the first correction process to the first image using correction data related to the first correction process, and applies the second correction process to the first image using correction data related to the second correction process, if the first correction process and the second correction process have not been applied to the first image.

17. The information processing apparatus according to claim 11, characterized in that the synthesis means generates a composite image to which only the second correction process is applied by applying a process in the opposite direction to the first correction process applied to the first image to the composite image obtained by synthesizing the first image to which the plurality of correction processes have been applied and the second image.

18. If the first image is a RAW video, the system has developing means that applies at least one of the plurality of correction processes to the RAW video to perform developing processing. The information processing apparatus according to claim 10, characterized in that the synthesis means synthesizes the frames of the video that have undergone the development process with the second image.

19. The information processing apparatus according to claim 10, characterized in that the second image is computer graphics.

20. A method for controlling an imaging device, The imaging device is An imaging means for capturing an image of a subject through an optical system, Image processing means capable of applying a plurality of correction processes relating to the optical aberrations of the optical system to the image obtained by the imaging means, The system includes recording means for recording the image and correction data for applying at least one of the plurality of correction processes to the image, The control method described above is A control method characterized by having the step of recording in the recording means correction data in which correction processing not applied to the image can be applied to the image, based on the format of the image to be recorded in the recording means and correction processing not applied to the image.

21. A method for controlling an information processing device, The first step is to obtain the first image, The process includes the steps of applying at least one of a plurality of correction processes related to optical aberrations to the first image, and compositing a pre-created second image onto the first image to which the plurality of correction processes have been applied, The control method is characterized in that, in the step of combining the first image and the second image, the unapplied correction processing among the plurality of correction processing that has not been applied to the first image is applied to the first image using correction data that can be applied to the first image.

22. A program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 9.

23. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 10 to 19.

24. An imaging device according to any one of claims 1 to 9, A system comprising the information processing device described in any one of claims 10 to 19.

Images