Information processing device and information processing method

Abstract

To make it possible to facilitate identifying what kind of RAW images are a plurality of RAW images included in one file.SOLUTION: An information processing device has acquisition means for acquiring a plurality of RAW images of different brightness and recording means for generating a file including the plurality of RAW images and information on the plurality of RAW images to record it to a recording medium.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing apparatus and an information processing method, and more particularly to a technique for synthesizing a plurality of images to obtain an image with a wide dynamic range. [Background technology]

[0002] DGO (Dual Gain Output) sensors are used as imaging elements in imaging devices such as digital single-lens reflex cameras, digital still cameras, and digital video cameras. DGO sensors amplify / attenuate the output signal from one pixel (photoelectric conversion element) with two gains in a column circuit to which the output signal is input, and can output two images with different brightness.

[0003] Now consider the case of HDR (High Dynamic Range) compositing. HDR compositing is an image process in which multiple SDR (Standard Dynamic Range) images with different brightness are combined to obtain an HDR image. The DGO sensor can output two images with different brightness with a single exposure. Therefore, when combining two images obtained by time-shared exposure (two exposures), a registration process for the two images is required, but when combining two images obtained by the DGO sensor, no registration process is required. In this way, the DGO sensor is well suited to HDR compositing.

[0004] Patent Document 1 discloses a technique for recording (saving) one file containing two RAW images on a storage medium. By using this technique, after shooting, it is possible to freely change image processing parameters such as exposure adjustment, white balance adjustment, optical correction, and noise reduction, develop the two RAW images, and then synthesize the two images obtained by the development process. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2005-323162 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the technique disclosed in Patent Document 1, it is not easy to identify what type of RAW images multiple RAW images contained in one file are.

[0007] An object of the present invention is to make it possible to easily identify what type of RAW images multiple RAW images contained in one file are. [Means for solving the problem]

[0008] The information processing device of the present invention is characterized by having an acquisition means for acquiring multiple RAW images having different brightness levels, and a recording means for generating a single file containing the multiple RAW images and information about the multiple RAW images and recording the file on a storage medium. Effect of the Invention

[0009] According to the present invention, it becomes possible to easily identify what type of RAW images multiple RAW images contained in one file are. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging apparatus. [Diagram 2] FIG. 2 is a block diagram showing a configuration of an imaging element. [Diagram 3] FIG. 2 is a circuit diagram showing a configuration of a pixel unit. [Figure 4] FIG. 1 is a block diagram showing a configuration related to HDR synthesis. [Diagram 5] 4 is a graph showing the relationship between the amount of light input to an imaging element and the brightness of a developed image. [Figure 6]5 is a flowchart showing a recording process according to the first embodiment. [Figure 7] FIG. 2 is a schematic diagram showing a file structure. [Figure 8] FIG. 2 is a schematic diagram showing ImageData. [Figure 9] 10 is a flowchart showing a reproduction process according to the second embodiment. [Figure 10] 13 is a flowchart showing a reproduction process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (First embodiment) A first embodiment of the present invention will be described below. Fig. 1 is a block diagram showing the configuration of an image capturing apparatus 100 according to the first embodiment.

[0012] The system control unit 11 controls the entire imaging device 100. The nonvolatile memory 12 is an electrically erasable and recordable memory, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory). Various information such as constants and programs for the operation of the system control unit 11 is stored in the nonvolatile memory 12. The system memory 13 is, for example, a RAM (Random Access Memory). Various information such as constants and variables for the operation of the system control unit 11 and programs read from the nonvolatile memory 12 is stored in the system memory 13. The system control unit 11 realizes each process described later by expanding the programs stored in the nonvolatile memory 12 into the system memory 13 and executing them.

[0013] The optical lens 101 forms an image of light from a subject on the image sensor 102. The image sensor 102 converts the light from the optical lens 101 into an electrical signal and outputs the electrical signal. The image sensor 102 is, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor 102 may be an image sensor that outputs an analog signal as a video signal. The image sensor 102 may perform analog-to-digital (AD) conversion internally and output digital data such as a low voltage differential signaling (LVDS) signal as a video signal.

[0014] FIG. 2 is a block diagram showing the configuration of the image sensor 102.

[0015] A timing pulse control section 201 controls the operation of the image sensor 102 by supplying an operating clock to each section of the image sensor 102 and by supplying a timing signal to each section.

[0016] The vertical scanning circuit 202 performs timing control for reading out a video signal from the pixel unit 203. The pixel unit 203 has a plurality of photoelectric conversion elements (a plurality of pixels) arranged two-dimensionally (for example, in a matrix). The photoelectric conversion elements convert light incident on the photoelectric conversion elements into a voltage (pixel signal voltage). The timing control performed by the vertical scanning circuit 202 is timing control for sequentially reading out pixel signal voltages from the plurality of photoelectric conversion elements in one frame period. In general, the pixel signal voltages are read out row by row from the top row to the bottom row of the pixel unit 203.

[0017] The column AMP (Amplifier) ​​204 electrically amplifies the video signal read out from the pixel unit 203. This increases the difference between the level of the video signal output from the column AMP 204 and the level of noise output from the column ADC (Analog-to-Digital Converter) 205. As a result, the signal-to-noise (SN) ratio of the video signal output from the column ADC is improved.

[0018] The timing pulse control unit 201 can change the gain (amplification factor) of the column AMP 204. The column AMP 204 has two input memories and can output two video signals with different gains. By using the two input memories, two video signals corresponding to the same time can be obtained as two video signals with different gains. Note that the column AMP 204 may be able to output three or more video signals corresponding to the same time.

[0019] The column ADC 205 converts an analog signal, which is a video signal output (read out) from the column AMP 204, into a digital signal (digital data) by AD conversion.

[0020] The horizontal transfer circuit 206 reads out the digitized video signal from the column ADC 205 , and outputs the video signal to the signal processing circuit 207 .

[0021] The signal processing circuit 207 is a circuit that performs digital processing (digital signal processing), performs digital processing on the video signal output from the horizontal transfer circuit 206, and outputs the digitally processed video signal to the external output circuit 208. For example, the signal processing circuit 207 can perform, as the digital processing, offset processing that adds a predetermined value and gain processing that multiplies a predetermined value. When the pixel unit 203 has a pixel area (light-shielded area) that is intentionally shaded from light, the signal processing circuit 207 may perform, as the digital processing, black level clamp processing using the video signal of the light-shielded area.

[0022] The external output circuit 208 has a serializer function and converts a multi-bit parallel signal, which is the video signal output from the signal processing circuit 207, into a serial signal. Then, the external output circuit 208 converts the obtained serial signal into a video signal of a predetermined format (for example, an LVDS signal), and outputs the video signal to the outside (for example, the image acquisition unit 103 in FIG. 1).

[0023] Returning to the description of FIG. 1, the image acquisition unit 103 performs various processes on the video signal output from the image sensor 102. For example, when AD conversion is not performed inside the image sensor 102, the image acquisition unit 103 has an analog front end and performs AD conversion on the video signal output from the image sensor 102. The image acquisition unit 103 may perform various processes such as removing fixed pattern noise of the image sensor 102 and black level clamping. The image acquisition unit 103 may separate the video signal output from the image sensor 102 into a video signal for recording and a control signal (for example, a signal for controlling the image sensor 102). In the first embodiment, the image acquisition unit 103 outputs the video signal for recording to the signal processing unit 104 and outputs the control signal to the exposure control unit 107.

[0024] The signal processing unit 104 performs various processes, including development, on the video signal (RAW image) output from the image acquisition unit 103. Hereinafter, the image after development is referred to as a developed image. Then, the signal processing unit 104 outputs the processed video signal (developed image) to the image synthesis unit 105. For example, the signal processing unit 104 performs at least one of pixel addition, noise reduction, gamma correction, knee correction, digital gain processing, and defect correction.

[0025] Furthermore, the image acquisition unit 103 and the signal processing unit 104 each have a memory circuit that stores setting values ​​required for various processes.

[0026] The image synthesis unit 105 acquires from the signal processing unit 104 a plurality of developed images (video signals of a plurality of developed images) corresponding to a plurality of (for example, two) RAW images output from the imaging element 102, respectively, and synthesizes the plurality of developed images. This generates a synthesized image (video signal of the synthesized image) having a different gradation from the plurality of developed images. For example, the image synthesis unit 105 performs image processing (HDR synthesis) to synthesize a plurality of SDR (Standard Dynamic Range) images having different brightness to obtain an HDR (High Dynamic Range) image. Then, the image synthesis unit 105 outputs the video signal of the synthesized image to the image recording / playback unit 106.

[0027] The image recording and playback unit 106 generates one file including a plurality of (for example, two) RAW images output from the imaging element 102, and records the generated file in a storage device that is an external device of the imaging device 100, or in a storage medium 110 provided in the imaging device 100. In the first embodiment, the image recording and playback unit 106 also stores the composite image output from the image composition unit 105 in the file. The image recording and playback unit 106 can also play back images (display images on a display unit, not shown) based on the file recorded in the storage device or storage medium 110. The storage medium 110 may be built in the imaging device 100, or may be detachable from the imaging device 100.

[0028] The exposure control unit 107 calculates an optimal exposure amount based on a signal (a control signal) output from the image acquisition unit 103, and outputs the calculation result to the image sensor control unit .

[0029] The image sensor control unit 108 controls the image sensor 102 in accordance with the optimal exposure amount output from the image sensor control unit 108 .

[0030] FIG. 3 is a circuit diagram showing the configuration of a portion of the column AMP204 corresponding to one column of the pixel unit 203. The circuit in FIG. 3 includes switches (SW) 301, SW302, capacitances (C) 303, C304, operational amplifiers (OP) 305, SW307, C308, and C309. SW301 is an operational amplifier. The ratio of the input capacitance and feedback capacitance connected to OP305 (input capacitance / feedback capacitance) is the gain (amplification rate) of OP305. For example, when SW301 is closed and SW302 is opened, C303 is connected to OP305 as an input capacitance, and when SW301 is opened and SW302 is closed, C304 is connected to OP305 as an input capacitance. When SW307 is opened, C306 is connected to OP305 as a feedback capacitance, and when SW307 is closed, the combined capacitance of C306 and C308 is connected to OP305 as a feedback capacitance. Therefore, the gain of the OP 305 can be switched (changed) by switching the open / closed states of the SWs 301, 302, and 307. The OP 305 amplifies the video signal (RAW image) read from the pixel unit 203 with a gain according to the open / closed states of the SWs 301, 302, and 307, and outputs the amplified video signal (RAW image) to the column ADC 205. By sequentially switching the gain of the OP 305, the brightness of the video signal (RAW image) read from the pixel unit 203 is sequentially changed with a plurality of gains, and a plurality of video signals with different gains (a plurality of RAW images with different brightnesses) can be obtained.

[0031] FIG. 4 is a block diagram showing a configuration related to the synthesis of a plurality of developed images (for example, HDR synthesis). Here, it is assumed that an H-developed image, which is a developed image obtained at a high gain higher than a normal gain, and an L-developed image, which is a developed image obtained at a low gain lower than a normal gain, are synthesized. The exposure correction unit 401 is a part of the signal processing unit 104, and adjusts the brightness of the H-developed image and the brightness of the L-developed image so that the brightness of the synthesized image increases linearly with an increase in the input light amount (actual brightness of the subject) of the image sensor 102. The image synthesis unit 402 is the image synthesis unit 105, and generates a synthesized image by synthesizing the H-developed image and the L-developed image after the brightness adjustment.

[0032] 5(A) to 5(C) are graphs in which the horizontal axis represents the amount of light input to the image sensor 102 and the vertical axis represents the brightness of the developed image.

[0033] The thick line in Fig. 5(A) indicates the correspondence relationship between the amount of light input to the image sensor 102 and the brightness of the H-developed image, and the thin line in Fig. 5(A) indicates the correspondence relationship between the amount of light input to the image sensor 102 and the brightness of the L-developed image. As shown in Fig. 5(A), the correspondence relationship of the H-developed image (thick line) and the correspondence relationship of the L-developed image (thin line) are significantly different. Therefore, the H-developed image and the L-developed image cannot be directly combined.

[0034] Therefore, the exposure correction unit 401 in Fig. 4 matches the brightness of the H-developed image with the brightness of the L-developed image. For example, as shown in Fig. 5(B), the exposure correction unit 401 applies (multiplies) a gain larger than 1 to the L-developed image to obtain an L2-developed image that matches the brightness of the H-developed image.

[0035] Then, the image synthesis unit 402 in Fig. 4 synthesizes the H-developed image and the L2-developed image. For example, as shown in Fig. 5(C), the image synthesis unit 402 replaces the blown-out areas of the H-developed image with the L2-developed image to generate a synthesized image.

[0036] The method of generating a composite image is not limited to the above method. For example, a gain smaller than 1 may be applied to the H-developed image to obtain an H2-developed image that matches the brightness of the L-developed image. Then, a composite image may be generated by replacing the black crushed area of ​​the L-developed image with the H2-developed image. The H2-developed image and the L-developed image may be synthesized. The brightness of both the H-developed image and the L-developed image may be adjusted, and the H-developed image and the L-developed image after the brightness adjustment may be synthesized. An appropriate-developed image, which is a developed image obtained with a normal gain, may be used instead of the H-developed image or the L-developed image. Three or more developed images corresponding to three or more RAW images may be synthesized. In addition, although an example of generating a composite image by synthesizing a plurality of developed images has been described, a composite image in RAW format may be generated by synthesizing a plurality of RAW images, and the composite image may be subjected to development processing. As long as a composite image having a different gradation from the plurality of images can be generated by synthesizing a plurality of images (a plurality of RAW images or a plurality of developed images), the method (synthesizing algorithm) is not particularly limited.

[0037] Fig. 6 is a flowchart showing the recording process of the imaging device 100. The recording process in Fig. 6 is realized by the system control unit 11 expanding a program stored in the non-volatile memory 12 into the system memory 13 and executing it. The recording process in Fig. 6 is performed in response to a shooting instruction from a user, for example.

[0038] In step S601, the system control unit 11 determines whether or not to perform HDR shooting to obtain an HDR image as a composite image. If HDR shooting is to be performed, the process proceeds to step S602, and if not (if SDR shooting is to be performed to obtain an SDR image as a composite image), the process proceeds to step S603.

[0039] In step S602, the system control unit 11 determines the gain (amplification rate, ISO sensitivity) of the image sensor 102 so as to obtain an H-RAW image and an L-RAW image. The H-RAW image is a RAW image for obtaining an H-developed image, and the L-RAW image is a RAW image for obtaining an L-developed image.

[0040] In step S602, the system control unit 11 determines the gain of the image sensor 102 so as to obtain an optimal RAW image and an L-RAW image. The optimal RAW image is a RAW image for obtaining the optimal RAW image.

[0041] The method of determining the gain is not limited to the above method. The system control unit 11 may determine the gain according to an instruction from a user, or may automatically determine the gain. The system control unit 11 may determine the gain according to the shooting mode. For example, the system control unit 11 may determine the gain so that an appropriate H-RAW image and L-RAW image are obtained in a still image shooting mode, and may determine the gain so that an H-RAW image and an L-RAW image are obtained in a video shooting mode.

[0042] In step S604, the system control unit 11 sets the gain determined in step S602 or step S603 in the image sensor 102, and acquires two RAW images with different gains (two RAW images with different brightness) using the image acquisition unit 103. When the process of step S602 is performed, an H-RAW image and an L-RAW image are acquired, and when the process of step S603 is performed, an appropriate RAW image and an L-RAW image are acquired. Note that the L-RAW image is a RAW image darker than the appropriate RAW image, and the H-RAW image is a RAW image brighter than the appropriate RAW image. Similarly, the L-developed image is a developed image darker than the appropriate-developed image, and the H-developed image is a developed image brighter than the appropriate-developed image.

[0043] In step S605, the system control unit 11 uses the signal processing unit 104 to perform development processing on the two RAW images acquired in step S604, thereby generating two developed images. Note that in step S605, the brightness adjustment described with reference to Figs. 4 and 5(B) is also performed. Hereinafter, the developed images generated in step S605 will be referred to as material images.

[0044] In step S606, the system control unit 11 uses the image synthesis unit 105 to synthesize the two material images generated in step S605, thereby generating a synthetic image.

[0045] In step S607, the system control unit 11 uses the image recording / playback unit 106 to encode the composite image generated in step S606. Hereinafter, the encoded composite image will be referred to as a display composite image (composite image for display). When SDR shooting is performed, for example, a display composite image in JPEG (Joint Photographic Experts Group) format is obtained. When HDR shooting is performed, for example, a HEVC (High Energy Video Conversion) format is obtained. A display composite image in the Efficiency Video Codec (EFC) format is obtained. The display composite image has the same resolution as the RAW image or the material images. The system control unit 11 may generate an LTHM (Learge Thumbnail) image or a THM (Thumbnail) image for simplified display by encoding and resizing the composite image using the image recording / playback unit 106.

[0046] In step S608, the system control unit 11 uses the image recording / playback unit 106 to encode the two material images generated in step S605. Hereinafter, the encoded material images are referred to as display material images (material images for display). The encoding format of the display material images may be the same as that of the display composite image, or may be different. If the encoding format of the display material images is the same as that of the display composite image, the display composite image and the display material images can be reproduced using the same process, thereby making the reproduction more efficient.

[0047] In step S609, the system control unit 11 uses the image recording / playback unit 106 to generate metadata of the file to be recorded in the storage medium 110. The system control unit 11 stores information of the two RAW images acquired in step S604 in the metadata. The information of the two RAW images is, for example, information that can identify whether each of the two RAW images is an H-RAW image, an L-RAW image, or an appropriate RAW image. Whether a RAW image is an H-RAW image, an L-RAW image, or an appropriate RAW image may be indicated by bit definition.

[0048] In step S610, the system control unit 11 uses the image recording / playback unit 106 to generate one file (RAW image file) including the two RAW images acquired in step S604 and the metadata generated in step S609. Then, the system control unit 11 uses the image recording / playback unit 106 to record the generated file in the storage medium 110. In the first embodiment, the system control unit 11 also stores the display composite image generated in step S607 and the two display material images generated in step S608 in a file to be recorded in the storage medium 110. The metadata is stored in, for example, a file header. The metadata may be stored in a MakerNote that is uniquely defined by the developer / manufacturer of the imaging device 100.

[0049] Fig. 7 is a schematic diagram showing the structure of a file recorded on storage medium 110. Although the container format (file format) is not particularly limited, the file in Fig. 7 has the ISO base media file format defined in ISO / IEC14496-12. Therefore, the file in Fig. 7 has a tree structure, with each node called a box. Each box can have multiple boxes as child elements.

[0050] A file 701 has an ftyp 702 at the beginning, followed by moov 703, uuid 709, Etc 708, and mdat 711. The ftyp 702 is a box that stores (describes) the file type. The moov 703 is a box that stores metadata. The uuid 709 is a user-defined box. The mdat 711 is a box that stores media data (image data). The Etc 708 is a miscellaneous box.

[0051] Moov 703 has, as its child elements, uuid 704, which is a user-defined box, and trak 707, which is a box that stores information referencing ImageData 712. uuid 704 has, as its child elements, MetaData 705, which is a box that stores metadata, and THM 706, which is a box that stores image data of a THM image. The metadata includes, for example, the creation date and time of the file, shooting conditions, information on whether shooting was performed in HDR or SDR, RAW image information (suitable RAW image / H-RAW image / L-RAW image), and other shooting information.

[0052] The uuid 709 has, as a child element, an LTHM 710 which is a box that stores image data of an LTHM image. The mdat 711 has, as a child element, an ImageData 712 which is a box that stores image data of a RAW image, a display source image, and a display composite image.

[0053] Image data stored in ImageData712, THM706, and LTHM710 differs between SDR and HDR shooting.

[0054] The file structure is not limited to the above structure. For example, a box different from the above box may be included in the file. At least one of the display composite image and the display material image may not be included in the file.

[0055] Fig. 8(A) is a schematic diagram showing ImageData 812 in the case where a still image has been shot in SDR. The ImageData 812 in Fig. 8(A) stores an appropriate RAW image 823, an L-RAW image 824, and RAW development parameters 825. The RAW development parameters are parameters used in the development process of a RAW image. The ImageData 812 in Fig. 8(A) further stores a display composite image 820, an appropriate display material image 821, and an L-display material image 822. The display composite image 820, the appropriate display material image 82 1 and L-display material image 822 are SDR images in JPEG format. Appropriate-display material image 821 is a display material image corresponding to the appropriate-RAW image, and is generated by encoding the appropriate-developed image. L-display material image 822 is a display material image corresponding to the L-RAW image, and is generated by encoding the L-developed image.

[0056] FIG. 8B is a schematic diagram showing ImageData812 when SDR shooting of a video is performed. In ImageData812 of FIG. 8B, an H-RAW image 827, an L-RAW image 824, and RAW development parameters 825 are stored. In ImageData812 of FIG. 8B, a display composite image 820, an H-display material image 826, and an L-display material image 822 are further stored. Like the display composite image 820 and the L-display material image 822, the H-display material image 826 is also an SDR image in JPEG format. The H-display material image 826 is a display material image corresponding to the H-RAW image, and is generated by encoding the H-developed image. A video file can be generated by storing data of multiple frames in one file using ImageData812 of FIG. 8B as data of one frame.

[0057] FIG. 8C is a schematic diagram showing ImageData 812 when HDR shooting of a still image is performed. In ImageData 812 in FIG. 8C, a suitable-RAW image 823, an L-RAW image 824, and RAW development parameters 825 are stored. In ImageData 812 in FIG. 8C, a display composite image 830, a suitable-display material image 831, and an L-display material image 832 are further stored. The display composite image 830 is an HDR image in the HEVC format, and the suitable-display material image 831 and the L-display material image 832 are SDR images in the HEVC format. The suitable-display material image 831 is a display material image corresponding to the suitable-RAW image, and is generated by encoding the suitable-developed image. The L-display material image 832 is a display material image corresponding to the L-RAW image, and is generated by encoding the L-developed image.

[0058] FIG. 8(D) is a schematic diagram showing ImageData812 in the case where HDR shooting of a video is performed. In ImageData812 in FIG. 8(D), an H-RAW image 827, an L-RAW image 824, and RAW development parameters 825 are stored. In ImageData812 in FIG. 8(D), a display composite image 830, an H-display material image 836, and an L-display material image 832 are further stored. Like the L-display material image 832, the H-display material image 836 is also an SDR image in the HEVC format. The H-display material image 836 is a display material image corresponding to the H-RAW image, and is generated by encoding the H-developed image. A video file can be generated by storing data of multiple frames in one file using ImageData812 in FIG. 8(D) as data of one frame.

[0059] As described above, according to the first embodiment, when multiple RAW images are stored in one file, information on the multiple RAW images is also stored in the file. This makes it easy to identify what kind of RAW images the multiple RAW images contained in the file are, based on the information contained in the file. Note that in the first embodiment, multiple developed images (multiple display material images) corresponding to the multiple RAW images are also stored in the file. Therefore, it is easy to identify what kind of RAW images the multiple RAW images contained in the file are, based on the multiple developed images.

[0060] Second embodiment A second embodiment of the present invention will be described below. Note that, in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0061] Fig. 9 is a flowchart showing the playback process of the imaging device 100. The playback process in Fig. 9 is realized by the system control unit 11 expanding a program stored in the non-volatile memory 12 into the system memory 13 and executing it. The playback process in Fig. 9 is performed in response to a playback instruction from a user, for example. Here, it is assumed that the file including a plurality of RAW images is a file including an appropriate RAW image and an L-RAW image.

[0062] In step S901, the system control unit 11 uses the image recording / playback unit 106 to read a file (RAW image file) from the storage medium 110, and determines whether the file contains multiple RAW images. Whether the file contains multiple RAW images can be determined based on information included in the file (e.g., MetaData 705 in FIG. 7). If the file contains multiple RAW images, the process proceeds to step S902, and if not (if the file contains only one RAW image), the process proceeds to step S903.

[0063] In step S902, the system control unit 11 uses the image recording / playback unit 106 to play (display) the display composite image. The display composite image may be included in a file, or may be generated at the timing of step S902. Since a file including multiple RAW images is likely to be a file intended for playing a composite image, the display composite image is played first. Although details will be described later, multiple images including the display composite image and multiple display material images can be played.

[0064] In step S903, the system control unit 11 reproduces (displays) a display developed image (developed image for display) generated by performing development processing on the RAW image included in the file, using the image recording / playback unit 106. The display developed image may be included in the file, or may be generated at the timing of step S903.

[0065] In step S904, the system control unit 11 determines whether or not a file forwarding operation (a user operation for switching the file to be played back) has been performed. If a file forwarding operation has been performed, the process proceeds to step S905; if not, the process proceeds to step S906.

[0066] In step S905, the system control unit 11 switches the file for which images are to be reproduced to another file, and then the process proceeds to step S901.

[0067] When a file forwarding operation is performed while a display material image is being played back, and the file after the file change contains multiple RAW images, the display material image of the file after the file change may be played back in step S802. At this time, a display material image corresponding to the same gain as the display material image played back before the file change may be played back. For example, if an L-display material image was played back before the file change, the L-display material image of the file after the file change may be played back. This allows the user to efficiently check images of the same type.

[0068] In the case of continuous shooting (continuous shooting), there is a higher possibility that the user would like to check the same type of image than in the case of single shooting (single shooting). Therefore, in step S802, in the case of continuous shooting, the display material image of the file after switching may be played back, and in the case of single shooting, the display composite image of the file after switching may be played back.

[0069] In step S906, the system control unit 11 determines whether or not an image forwarding operation (a user operation for switching the image to be played back without switching the file) has been performed. If a forwarding operation has been performed, the process proceeds to step S907; if not, the process proceeds to step S904.

[0070] In step S907, the system control unit 11 plays (displays) the appropriate display material image using the image recording / playback unit 106. The appropriate display material image may be included in a file, or may be generated at the timing of step S907.

[0071] In step S908, the system controller 11 determines whether or not a file transfer operation has been performed. If a file transfer operation has been performed, the process proceeds to step S905, and if not, the process proceeds to step S909.

[0072] In step S909, the system control unit 11 determines whether or not an image forwarding operation has been performed. If an image forwarding operation has been performed, the process proceeds to step S910, and if not, the process proceeds to step S904.

[0073] In step S910, the system control unit 11 plays (displays) the L-display material image using the image recording / playback unit 106. The L-display material image may be included in a file, or may be generated at the timing of step S910.

[0074] In step S911, it is determined whether or not a file forwarding operation has been performed. If a file forwarding operation has been performed, the process proceeds to step S905, and if not, the playback process in FIG.

[0075] It should be noted that, if an image forwarding operation is performed again after step S910, the system control unit 11 plays (displays) the display composite image using the image recording / playback unit 106. In this manner, if a file contains multiple RAW images, the image to be played is switched between multiple images including the display composite image and multiple display material images in response to an image forwarding operation.

[0076] As described above, according to the second embodiment, it is possible to reproduce (display) a plurality of developed images (a plurality of display material images) corresponding to a plurality of RAW images included in a file. This allows the user to easily identify what kind of RAW images the plurality of RAW images included in the file are.

[0077] Although an example in which the imaging device 100 performs the reproduction process has been described, the reproduction process may be performed by a device (electronic device) separate from the imaging device 100. In addition, when a file contains multiple RAW images, information on the multiple RAW images (for example, multiple icons each showing the multiple RAW images) may be displayed. When a file contains multiple RAW images, multiple developed images corresponding to the multiple RAW images may be displayed side by side in response to a predetermined user operation. For example, an appropriate-display material image and an L-display material image may be displayed side by side. When a file contains multiple RAW images, a composite image and multiple developed images corresponding to the multiple RAW images may be displayed side by side in response to a predetermined user operation. For example, a display composite image, an appropriate-display material image, and an L-display material image may be displayed side by side in the left-right direction.

[0078] (Third embodiment) The third embodiment of the present invention will be described. Note that, in the following, the description of the same points as the first and second embodiments (for example, the same configurations and processes as the first and second embodiments) will be omitted, and only the points different from the first and second embodiments will be described.

[0079] Fig. 10 is a flowchart showing the playback process of the imaging device 100. The playback process in Fig. 10 is realized by the system control unit 11 expanding a program stored in the non-volatile memory 12 into the system memory 13 and executing it. The playback process in Fig. 10 is performed in response to a playback instruction from a user, for example. Here, it is assumed that the file including multiple RAW images is a file including an appropriate RAW image and an L-RAW image.

[0080] 9, in step S1001, the system control unit 11 uses the image recording / playback unit 106 to read a file (RAW image file) from the storage medium 110, and determines whether the file contains multiple RAW images. If the file contains multiple RAW images, the process proceeds to step S1002, and if not (if the file contains only one RAW image), the process proceeds to step S1003.

[0081] In step S1002, similarly to step S902, the system control unit 11 uses the image recording / playback unit 106 to play (display) the display composite image.

[0082] In step S1003, similarly to step S903, the system control unit 11 uses the image recording / playback unit 106 to play back (display) the displayed developed image.

[0083] In step S1004, the system control unit 11 determines whether or not a redevelopment operation (a user operation for performing development processing (redevelopment processing) using RAW development parameters newly specified by the user) has been performed. If a redevelopment operation has been performed, the process proceeds to step S1011; if not, the process proceeds to step S1005.

[0084] In step S1005, similarly to step S906, the system control unit 11 determines whether or not an image forwarding operation has been performed. If an image forwarding operation has been performed, the process proceeds to step S1006, and if not, the process proceeds to step S1004.

[0085] In step S1006, similarly to step S907, the system control unit 11 uses the image recording / playback unit 106 to play (display) the appropriate display material image.

[0086] In step S1007, the system control unit 11 determines whether or not a redevelopment operation has been performed. If a redevelopment operation has been performed, the process proceeds to step S1011, and if not, the process proceeds to step S1008.

[0087] In step S1008, similarly to step S909, the system control unit 11 determines whether or not an image forwarding operation has been performed. If an image forwarding operation has been performed, the process proceeds to step S1009, and if not, the process proceeds to step S1004.

[0088] In step S1009, similarly to step S910, the system control unit 11 uses the image recording / playback unit 106 to play (display) the L-display material image.

[0089] In step S1010, the system control unit 11 determines whether or not a redevelopment operation has been performed. If a redevelopment operation has been performed, the process proceeds to step S1011, and if not, the reproduction process of FIG.

[0090] In step S1011, the system control unit 11 performs redevelopment processing of the RAW image using the newly specified RAW development parameters. The redevelopment processing is performed using the signal processing unit 104. For example, when a redevelopment operation is performed while a display composite image is being played back, redevelopment processing of multiple RAW images (for example, redevelopment processing of an appropriate RAW image and an L-RAW image) is performed. When a redevelopment operation is performed while a display material image is being played back, the RAW image corresponding to the display material image is redeveloped. For example, when a redevelopment operation is performed while an appropriate-display material image is being played back, the appropriate-RAW image is redeveloped. When a redevelopment operation is performed while an L-display material image is being played back, the L-RAW image is redeveloped. When a file contains only one RAW image, the one RAW image is redeveloped.

[0091] In step S1012, the system control unit 11 determines whether or not redevelopment processing has been performed on multiple RAW images in step S1011. If redevelopment processing has been performed on multiple RAW images, the system control unit 11 proceeds to step S1013, and if not (if redevelopment processing has been performed on one RAW image), the system control unit 11 proceeds to step S1014.

[0092] In step S1013, similarly to step S606 in FIG. 6, the system control unit 11 uses the image synthesis unit 105 to synthesize the multiple developed images (multiple material images) generated in step S1011, thereby generating a composite image.

[0093] In step S1014, similarly to steps S607 and S608 in FIG. 6, the system control unit 11 encodes the image using the image recording and playback unit 106. This encoded image is hereinafter referred to as the redeveloped image. If multiple RAW images have been redeveloped, the composite image generated in step S1013 is encoded, and if not (if one RAW image has been redeveloped), the developed image generated in step S1011 is encoded. Then, the system control unit 11 updates the image to be played back to the redeveloped image using the image recording and playback unit 106.

[0094] In step S1015, the system control unit 11 updates the metadata of the file using the image recording / playback unit 106. For example, the system control unit 11 adds information about the redevelopment process (for example, the execution date and time of the redevelopment process) to the metadata.

[0095] In step S1016, the system control unit 11 uses the image recording / playback unit 106 to store the redeveloped image (and the RAW development parameters corresponding to the redeveloped image) in a file. The redeveloped image may be added to the file, or the image stored in the file may be replaced with the redeveloped image. For example, when a plurality of RAW images are redeveloped, the display composite image stored in the file may be replaced with the redeveloped image. When only the suitable RAW image is redeveloped, the suitable display material image stored in the file may be replaced with the redeveloped image. When only the L-RAW image is redeveloped, the L-display material image stored in the file may be replaced with the redeveloped image.

[0096] As described above, according to the third embodiment, an image that the user desires can be obtained by the redevelopment process.

[0097] The various controls described above as being performed by the system control unit 11 may be performed by at least one piece of hardware (for example, at least one processor and / or at least one circuit). One piece of hardware may control the entire device, or multiple pieces of hardware may share the processing to control the entire device.

[0098] In addition, although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0099] In the above-mentioned embodiment, the present invention is applied to an imaging device (digital camera), but the present invention is not limited to an imaging device and can be applied to any information processing device (electronic device) capable of acquiring a RAW image. For example, the present invention can be applied to a personal computer, a PDA, a mobile phone terminal, a portable image viewer, a printer device, a digital photo frame, a music player, a game machine, and an electronic book reader. The present invention can also be applied to a video player, a display device (including a projection device), a tablet terminal, a smartphone, an AI speaker, a home appliance device, and an in-vehicle device.

[0100] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0101] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring a plurality of RAW images having different brightness levels; a recording means for generating one file including the plurality of RAW images and information about the plurality of RAW images and recording the file on a storage medium; 13. An information processing device comprising: (Configuration 2) The plurality of RAW images are generated by changing the brightness of one RAW image with a plurality of gains. 2. The information processing device according to configuration 1. (Configuration 3) The method further includes a developing unit that performs a development process on the plurality of RAW images to generate a plurality of developed images, The recording means generates the file including the plurality of RAW images, information about the plurality of RAW images, and the plurality of developed images. 3. The information processing device according to configuration 1 or 2. (Configuration 4) a synthesis unit for generating a synthetic image having a different gradation from the plurality of developed images by synthesizing the plurality of RAW images or the plurality of developed images, The recording means generates the file including the plurality of RAW images, information about the plurality of RAW images, and the composite image. 4. The information processing device according to configuration 3. (Configuration 5) the plurality of developed images are SDR images; The composite image is an HDR image. 5. The information processing device according to configuration 4. (Configuration 6) The recording means stores, as the file, a first RAW image, a second RAW image that is darker than the first RAW image, and a file containing information about the first RAW image and the second RAW image; or the second RAW image, a third RAW image that is brighter than the first RAW image, and a file containing information on the second RAW image and the third RAW image; Generate 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The apparatus further includes a playback means for playing back an image based on the file recorded on the storage medium, The regeneration means comprises: determining whether a file to be reproduced contains a plurality of RAW images based on information contained in the file; If the target file contains multiple RAW images, first (1) A composite image that is generated by combining multiple RAW images included in the target file and has a different gradation from the multiple RAW images; and (2) A plurality of developed images generated by subjecting a plurality of RAW images contained in the target file to a development process; and Play one of several images, including 7. The information processing device according to any one of configurations 1 to 6, wherein the image to be reproduced is switched among the plurality of images in response to a first user operation. (Configuration 8) When the target file contains a plurality of RAW images, the reproduction means first reproduces the composite image. 8. The information processing device according to configuration 7. (Configuration 9) The regeneration means comprises: in response to a second user operation, switching the target file to another file; When the second user operation is performed in a state where a developed image generated by performing development processing on a RAW image included in the file before the switching is being played back, and the file after the switching includes a plurality of RAW images, first, the developed image generated by performing development processing on the RAW image included in the file after the switching is played back. 8. The information processing device according to configuration 7. (Configuration 10) The reproduction means updates the image to be reproduced to an image generated using newly designated development parameters in response to a third user operation. 10. The information processing device according to any one of configurations 7 to 9. (Configuration 11) The reproduction means, in response to the third user operation while the composite image is being reproduced, updates the image to be reproduced to a new composite image generated using the newly specified development parameters. 11. The information processing device according to configuration 10. (Configuration 12) The reproduction means, in response to the third user operation while the developed image is being reproduced, updates the image to be reproduced to a new developed image generated using the newly specified development parameters. 12. The information processing device according to configuration 10 or 11. (Configuration 13) When the target file contains a plurality of RAW images, the playback means controls to display information on the plurality of RAW images. 13. The information processing device according to any one of configurations 7 to 12. (Configuration 14) When the target file includes a plurality of RAW images, the playback means performs control to display the plurality of developed images side by side in response to a fourth user operation. 14. The information processing device according to any one of configurations 7 to 13. (Configuration 15) When the target file includes a plurality of RAW images, the reproduction means performs control in response to a fourth user operation so as to display the composite image and the plurality of developed images side by side. 14. The information processing device according to any one of configurations 7 to 13. (Configuration 16) a playback means for playing back an image based on a file recorded on a storage medium; The regeneration means comprises: If the file you want to play contains multiple RAW images, first (1) A composite image that is generated by combining multiple RAW images included in the target file and has a different gradation from the multiple RAW images; and (2) A plurality of developed images generated by subjecting a plurality of RAW images contained in the target file to a development process; and Play one of several images, including An information processing apparatus which switches an image to be reproduced among the plurality of images in response to a predetermined user operation. (Method 1) obtaining a plurality of RAW images having different brightness levels; generating one file including the plurality of RAW images and information about the plurality of RAW images and recording the file on a storage medium; 13. An information processing method comprising: (Method 2) reproducing an image based on the file recorded on the storage medium; Steps for switching the image to be played having If the file you want to play contains multiple RAW images, first (1) A composite image that is generated by combining multiple RAW images included in the target file and has a different gradation from the multiple RAW images; and (2) A plurality of developed images generated by subjecting a plurality of RAW images contained in the target file to a development process; and Any of the multiple images including In response to a predetermined user operation, the image to be reproduced is switched among the plurality of images. 23. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 16. (medium) 17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 16. [Explanation of symbols]

[0102] 100: Imaging device 11: System control section 103: Image acquisition unit 106: Image recording and playback unit

Claims

1. an acquisition means for acquiring a plurality of RAW images having different brightness levels; an image processing means for generating a plurality of developed images by subjecting the plurality of RAW images to development processing; a recording control means for generating a single file including the plurality of RAW images, information on the plurality of RAW images, and the plurality of developed images, and recording the file on a storage medium; and the image processing means is capable of generating a composite image having a different gradation from the plurality of developed images by performing a synthesis process on the plurality of RAW images or the plurality of developed images, The recording control means records the composite image after the combining process and the developing process in the one file, but does not record the composite RAW image generated by the combining process of the plurality of RAW images.

1. An information processing device comprising:

2. The plurality of RAW images are generated by changing the brightness of one RAW image with a plurality of gains.

2. The information processing apparatus according to claim 1, wherein:

3. A control means for determining the plurality of gains; A determination means for determining whether the image is HDR-captured; and When the determination means determines that HDR shooting is being performed, the control means determines a gain for acquiring a bright image and a gain for acquiring a dark image as the plurality of gains, and when the determination means determines that HDR shooting is not being performed, the control means determines a gain for acquiring a dark image and a gain for acquiring an image of normal brightness as the plurality of gains.

2. The information processing apparatus according to claim 1, wherein:

4. A determination means for determining whether or not an HDR image has been taken and When the determination means determines that the image was shot using HDR, the recording control means records, as the plurality of RAW images, a RAW image that is brighter than an appropriate brightness and a RAW image that is darker than an appropriate brightness in the one file; and when the determination means determines that the image was not shot using HDR, the recording control means records, as the plurality of RAW images, a RAW image that is brighter than an appropriate brightness and a RAW image that is darker than the appropriate brightness in the one file.

2. The information processing apparatus according to claim 1, wherein:

5. The recording control means records the composite image in the one file as a thumbnail image or a display image.

2. The information processing apparatus according to claim 1, wherein:

6. the plurality of developed images are SDR images; The composite image is an HDR image 2. The information processing apparatus according to claim 1, wherein:

7. A reproducing means for reproducing an image based on the file recorded on the storage medium. and The regeneration means determining whether a file to be reproduced contains a plurality of RAW images based on information contained in the file; If the target file contains multiple RAW images, first: (1) A composite image having a different gradation from the plurality of RAW images included in the target file, which is generated by a development process and a composition process using the plurality of RAW images included in the target file; (2) A plurality of developed images generated by performing development processing on a plurality of RAW images included in the target file; Play one of several images, including 2. The information processing apparatus according to claim 1, wherein the image to be played back is switched among the plurality of images in response to a first user operation.

8. When the target file contains a plurality of RAW images, the reproduction means first reproduces the composite image.

8. The information processing apparatus according to claim 7,

9. The regeneration means Switching the target file to another file in response to a second user operation; When the second user operation is performed in a state where a developed image generated by performing a development process on a RAW image included in the file before switching is being played back, and when a plurality of RAW images are included in the file after switching, first, the developed image generated by performing a development process on the RAW image included in the file after switching is played back.

8. The information processing apparatus according to claim 7,

10. The reproduction means, in response to a third user operation, updates the image to be reproduced to an image generated using newly designated development parameters.

8. The information processing apparatus according to claim 7,

11. The reproduction means, in response to the third user operation while the composite image is being reproduced, updates the image to be reproduced to a new composite image generated using the newly specified development parameters.

11. The information processing apparatus according to claim 10,

12. The reproduction means, in response to the third user operation while the developed image is being reproduced, updates the image to be reproduced to a new developed image generated using the newly specified development parameters.

11. The information processing apparatus according to claim 10,

13. When the target file contains a plurality of RAW images, the playback means controls the display so as to display information about the plurality of RAW images.

8. The information processing apparatus according to claim 7,

14. When the target file contains a plurality of RAW images, the playback means performs control to display the plurality of developed images side by side in response to a fourth user operation.

8. The information processing apparatus according to claim 7,

15. When the target file contains a plurality of RAW images, the playback means performs control in response to a fourth user operation to display the composite image and the plurality of developed images side by side.

8. The information processing apparatus according to claim 7,

16. a playback means for playing back images based on the files recorded on the storage medium; The regeneration means If the file you want to play contains multiple RAW images, first (1) A composite image having a different gradation from the plurality of RAW images included in the target file, which is generated by a development process and a composition process using the plurality of RAW images included in the target file; (2) A plurality of developed images generated by performing development processing on a plurality of RAW images included in the target file; Play one of several images, including An information processing device that switches the image to be played back among the plurality of images in response to a predetermined user operation.

17. an acquisition step of acquiring a plurality of RAW images having different brightness levels; an image processing step of generating a plurality of developed images by subjecting the plurality of RAW images to development processing; a recording control step of generating one file including the plurality of RAW images, information on the plurality of RAW images, and the plurality of developed images, and recording the file on a storage medium; and In the image processing step, a composite image having a different gradation from the plurality of developed images can be generated by performing a synthesis process on the plurality of RAW images or the plurality of developed images, In the recording control step, a composite image after the combining process and the developing process is recorded in the one file, but a composite RAW image generated by the combining process of the plurality of RAW images is not recorded.

1. An information processing method comprising:

18. Reproducing an image based on the file recorded on the storage medium; Steps for switching the image to be played and If the file you want to play contains multiple RAW images, first (1) A composite image having a different gradation from the plurality of RAW images included in the target file, which is generated by a development process and a composition process using the plurality of RAW images included in the target file; (2) A plurality of developed images generated by performing development processing on a plurality of RAW images included in the target file; One of the multiple images including In response to a predetermined user operation, the image to be played is switched among the plurality of images.

1. An information processing method comprising:

19. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 16.

20. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 16.

Images