Image processing device, imaging device, control method, and program

Abstract

This technology provides the ability to associate appropriate authenticity assurance information with single-exposure composite images. [Solution] The present invention provides an image processing apparatus comprising association means for associating a first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image, with the first image when the first image is either a first type image or a second type image, wherein the first type image is a non-composite image captured in a single exposure, and the second type image is a composite image generated by combining a plurality of non-composite images captured in a common single exposure.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program.

Background Art

[0002] Conventionally, in order to guarantee that an image captured by a digital camera has not been tampered with, a digital camera having a function of adding (associating) true guarantee information as verification data has been proposed.

[0003] For example, Patent Document 1 discloses a technique of embedding information (hash and signature value) for prohibiting tampering in an image file when shooting is performed in an image tampering prohibition mode.

[0004] In recent years, a technical standardization organization called C2PA has been developed to develop technical specifications that can guarantee the origin and reliability of content, so that publishers, creators, and consumers can track the origin of media. The need for true guarantee has been attracting attention again. In the C2PA standard, a Manifest can be saved in an image generated at the time of shooting or editing as C2PA data. In the Manifest, an Actor (the name of the camera or software that generated the image), a hash value, a claim signature (digital signature), a thumbnail image, etc. can be stored.

[0005] In addition, an imaging device having two column circuits for an output signal from a unit pixel and making the gain of an amplification unit in the column circuit different between the column circuits is known, and two images with different gains can be output. In this specification, such an imaging device is called a Dual Gain Output (DGO) imaging device. The DGO imaging device can output two images (High gain image and Low gain image) to which different gains are applied by performing shooting with one exposure.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2008-005421 [Overview of the project] [Problems that the invention aims to solve]

[0007] Some recent smartphones and other imaging devices have a function that captures multiple images, combines them according to some algorithm to generate a composite image, and records the composite image as a single photograph. Examples of such functions include HDR multi-shot, panoramic shooting, and multiple exposure shooting.

[0008] While such composite images are not necessarily tampered with, they may contain scenes that would be impossible to capture in a single exposure. Therefore, conventional technology does not provide authenticity assurance information for composite images.

[0009] Furthermore, a technique is known for generating a composite image with an expanded dynamic range by combining two images with different gains, captured in a single exposure using a DGO image sensor. A composite image with an expanded dynamic range, generated by combining two images with different gains captured in a single exposure using a DGO image sensor (sometimes referred to as a "DGO composite image"), is a type of composite image. Therefore, conventional techniques do not add authenticity assurance information to DGO composite images.

[0010] However, the exposure involved in a DGO composite image is only one exposure common to the two images used to generate the DGO composite image. Therefore, unlike composite images obtained by multiple exposures, etc. (sometimes referred to as "multiple exposure composite images"), the possibility of the phenomenon where "scenes that could not be captured in an image taken with a single exposure are recorded in the composite image" is low in DGO composite images. More generally speaking, not limited to DGO images, the possibility of the phenomenon where "scenes that could not be captured in an image taken with a single exposure are recorded in the composite image" is low in composite images generated by combining multiple non-composite images taken with a common single exposure (sometimes referred to as "single exposure composite images") is low.

[0011] This invention has been made in view of these circumstances and aims to provide a technology that enables the association of appropriate authenticity assurance information with a single-exposure composite image. [Means for solving the problem]

[0012] To solve the above problems, the present invention provides When the first image is either a first type image or a second type image, the system includes association means for associating the first image with first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image. The aforementioned first type of image is a non-composite image captured in a single exposure. The second type of image described above is a composite image generated by combining multiple non-composite images taken in a single common exposure. The present invention provides an image processing apparatus characterized by the following features. [Effects of the Invention]

[0013] According to the present invention, it is possible to associate appropriate authenticity assurance information with a single-exposure composite image.

[0014] In addition, other features and advantages of the present invention will become more apparent from the accompanying drawings and the description in the following embodiments for carrying out the invention.

Brief Description of the Drawings

[0015] [Figure 1A] A block diagram showing a configuration example of the digital camera 100. [Figure 1B] A diagram showing the configuration of the imaging device 13. [Figure 1C] A conceptual diagram of a circuit extracting one column from the column AMPs 124. [Figure 2A] A diagram showing a configuration example of a plurality of exposure synthesis image files. [Figure 2B] A diagram showing another configuration example of a plurality of exposure synthesis image files. [Figure 2C] A diagram showing a configuration example of a non-synthesis image file captured by a single exposure. <0,000,080> [Figure 2D] A diagram showing a configuration example of a DGO synthesis image file? [Figure 3] A flowchart of a process for adding true guarantee information to an image.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted. <,000,091> [First Embodiment]<,000,093> FIG. 1A is a block diagram showing a configuration example of the digital camera 100. The digital camera 100 is an example of an imaging device including an image processing device. The imaging device may be, for example, a smartphone. Also, a personal computer (PC) without a camera can play the role of the image processing device of the present embodiment.

[0018] The barrier 10 is a protective member that covers the imaging unit including the photographing lens 11 of the digital camera 100 to prevent dirt and damage to the imaging unit, and its operation is controlled by the barrier control unit 43. The photographing lens 11 forms an optical image on the imaging surface of the image sensor 13. The shutter 12 has an aperture function. The image sensor 13 is composed of, for example, a CCD, a CMOS sensor, etc., and converts the optical image formed on the imaging surface by the photographing lens 11 via the shutter 12 into an electrical signal.

[0019] The A / D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal converted by the A / D converter 15 is written into the memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on the information at the time of shooting and written into the memory 25. The development parameters are composed of various parameters used in image processing for recording images in the JPEG format, such as exposure settings, white balance, color space, contrast, etc.

[0020] The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50, and supplies clock signals and control signals to the image sensor 13, the A / D converter 15, and the D / A converter 21.

[0021] The image processing unit 20 performs various image processes such as predetermined pixel interpolation processing, color conversion processing, correction processing, resizing processing, and image composition processing on the data from the A / D converter 15 or the data from the memory control unit 22. In addition, the image processing unit 20 performs predetermined image processing and arithmetic processing using the obtained image data, and provides the obtained arithmetic result to the system control unit 50. The system control unit 50 controls the exposure control unit 40 and the focus control unit 41 based on the provided arithmetic result to realize autofocus (AF) processing, automatic exposure (AE) processing, and flash pre-emission (EF) processing.

[0022] Furthermore, the image processing unit 20 performs predetermined calculations using the image data obtained from the capture, and also performs auto white balance (AWB) processing based on the obtained calculation results. In addition, the image processing unit 20 reads the image data stored in the memory 25 and performs compression or decompression processing using JPEG, MPEG-4 AVC, High Efficiency Video Coding (HEVC), or a lossless compression method for uncompressed RAW data. Finally, the image processing unit 20 writes the processed image data to the memory 25.

[0023] Furthermore, the image processing unit 20 performs predetermined calculations using the image data obtained by capturing, and performs various image data editing processes. Specifically, it can perform cropping, which adjusts the display range and size of the image by hiding unnecessary parts around the image data, and resizing, which changes the size of the image data or screen display elements by enlarging or reducing them. In addition, the image processing unit 20 can perform RAW development, which adds image processing such as color conversion to data that has been compressed or decompressed using a lossless compression method for uncompressed RAW data, and converts it into JPEG or HEVC format data to create image data. Furthermore, the image processing unit 20 can perform video extraction, which extracts a specified frame from a video format such as MPEG-4, converts it into JPEG format data, and saves it.

[0024] Furthermore, the image processing unit 20 performs predetermined calculations using the image data and performs image comparison processing of various image data. Specifically, it performs decompression processing according to the compression method of the image data to be compared and compares the decompressed images. The image processing unit 20 can determine whether the images match or not, and how much the difference is.

[0025] Furthermore, the image processing unit 20 also performs processing such as overlaying an On-Screen Display (OSD) such as a menu or arbitrary characters to be displayed on the display unit 23, along with the image data for display.

[0026] Furthermore, the image processing unit 20 uses the input image data and distance information to the subject obtained from the image sensor 13 during shooting to detect subjects present in the image data and performs subject detection processing to detect the subject's region. The detectable information includes region information such as position and size within the image, as well as detection information such as tilt and certainty.

[0027] Furthermore, the image processing unit 20 includes a synthesis processing circuit for synthesizing multiple image data. In this embodiment, the image processing unit 20 may synthesize images by overwriting pixels, or by weighted addition. Weighted addition can produce an image in which the background is transparent. The image processing unit 20 can also perform a comparative light synthesis process or a comparative dark synthesis process, which selects the image with the brightest or darkest value in each region of the image data to be synthesized, and synthesizes the selected image for each pixel to generate a single image data.

[0028] The memory control unit 22 controls the A / D converter 15, timing generator 14, image processing unit 20, image display memory 24, D / A converter 21, and memory 25. The RAW image data generated by the A / D converter 15 is written to the image display memory 24 or memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22.

[0029] Image data for display written to the image display memory 24 is displayed on the display unit 23, which is composed of a TFT LCD or the like, via the D / A converter 21. By sequentially displaying the image data obtained by imaging using the display unit 23, it is possible to realize an electronic viewfinder function that displays live images.

[0030] Memory 25 has sufficient storage capacity to store a predetermined number of still images and a predetermined amount of video footage, and stores the captured still images and video footage. Memory 25 can also be used as a workspace for the system control unit 50.

[0031] The exposure control unit 40 controls the shutter 12, which has an aperture function. The exposure control unit 40 also has a flash metering function by working in conjunction with the flash 44. The focus control unit 41 adjusts the focus by driving a focus lens (not shown) included in the photographic lens 11 based on instructions from the system control unit 50. The zoom control unit 42 controls zooming by driving a zoom lens (not shown) included in the photographic lens 11. The flash 44 has an AF assist light projection function and a flash metering function.

[0032] The system control unit 50 controls the entire digital camera 100. The non-volatile memory 51 is an electrically erasable and recordable non-volatile memory, such as an EEPROM. The non-volatile memory 51 stores not only programs but also map information and other data.

[0033] The shutter switch 61 (SW1) turns ON during the operation of the shutter button 60 and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) turns ON when the operation of the shutter button 60 is completed and instructs the start of a series of shooting operations including exposure processing, development processing, and recording processing. In the exposure processing, the system control unit 50 controls the system to write the signal read from the image sensor 13 to the memory 25 as RAW image data via the A / D converter 15 and the memory control unit 22. In the development processing, the system control unit 50 uses calculations in the image processing unit 20 and the memory control unit 22 to develop the RAW image data written to the memory 25 and controls the system to write it to the memory 25 as image data. In the recording processing, the system control unit 50 reads the image data from the memory 25, compresses it using the image processing unit 20, stores the compressed image data in the memory 25, and then controls the system to write it to the external recording medium 91 via the card controller 90.

[0034] The control unit 63 is equipped with various buttons and touch panels. For example, the control unit 63 includes a power button, a menu button, a mode switch for switching between shooting mode / playback mode / other special shooting modes, a directional pad, a set button, a macro button, and a multi-screen playback page change button. In addition, the control unit 63 also includes, for example, a flash setting button, a single-shot / continuous-shot / self-timer switching button, a menu navigation + (plus) button, a menu navigation - (minus) button, a shooting quality selection button, an exposure compensation button, and a date / time setting button.

[0035] The metadata generation and analysis unit 70 generates various metadata, such as Exchangeable image file format (Exif) information, to be attached to the image data when recording image data to the external recording medium 91, based on the information at the time of shooting. Furthermore, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata attached to the image data. Examples of metadata include various setting information at the time of shooting, image data information related to the image data, and characteristic information of the subject contained in the image data. Additionally, when recording moving image data, the metadata generation and analysis unit 70 can generate and attach metadata to each frame.

[0036] The power supply 80 consists of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter, etc. The power control unit 81 supplies power from the power supply 80 to each part of the digital camera 100.

[0037] The card controller 90 transmits and receives data to and from an external recording medium 91, such as a memory card. The external recording medium 91 is, for example, a memory card, and records images (still images, videos) taken by the digital camera 100.

[0038] The communication unit 71 has a communication circuit for transmitting and receiving data. Specifically, the communication circuit may be configured to perform wireless communication such as Wi-Fi or Bluetooth (registered trademark), or it may be configured to perform wired communication such as Ethernet or USB.

[0039] The hash value generation unit 72 generates (calculates) a hash value by executing a hash function on various data (e.g., image data or metadata) input via the system control unit 50. Algorithms for generating hash values ​​include SHA256, SHA384, and SHA512. Note that the system control unit 50 may generate the hash value instead of the hash value generation unit 72. Furthermore, the hash value generation unit 72 may generate a hash value by executing a hash function on the entire image file, rather than just the image data.

[0040] The signature generation / verification unit 73 generates and verifies signature information necessary for determining authenticity. When an image is created, the signature generation / verification unit 73 generates signature information using the hash value of the image data generated by the hash value generation unit 72 and the signature generation key (private key), and records it in the image file as authenticity guarantee information. When image tampering is detected, the signature generation / verification unit 73 determines whether or not the image has been tampered with by verifying the hash value of the image data to be verified, generated by the hash value generation unit 72, and the signature recorded as authenticity guarantee information using the public key. Examples of algorithms for generating and verifying signature information include ECDSA, RSASSA-PSS, and EdDSA. Note that the system control unit 50 may perform the role of the signature generation / verification unit 73 instead of the signature generation / verification unit 73.

[0041] The configuration of the image sensor 13 will now be explained with reference to Figure 1B. The image sensor 13 is configured to generate multiple images with different gains applied in a single common exposure. In Figure 1B, the timing pulse control unit 121 controls the operation of the image sensor 13 by supplying an operating clock (CLK) to each block of the image sensor 13 and supplying timing signals to each block.

[0042] The vertical scanning circuit 122 performs timing control to sequentially read out the pixel signal voltages of the pixel section 123, in which multiple pixels are arranged in two dimensions, within one frame. Generally, the video signal is read out sequentially row by row within one frame, from the top row to the bottom row.

[0043] The pixel unit 123 is a photoelectric conversion element that converts incident light into photoelectric energy and outputs a voltage corresponding to the amount of incident light. The pixel unit 123 converts the captured light into electric charge and stores the charge in a Floating Diffusion (FD) capacitor. The size of the FD capacitor is changeable, and the signal-to-noise ratio (SNR) can be improved by changing the capacitor size according to the ISO sensitivity. Generally, the capacitor is set to "large" at low ISO sensitivities and to "small" at high ISO sensitivities. When outputting images with two different gains, as described later, the capacitor that stores the charge is common to both gains. Furthermore, the settable capacitor size is not limited to just "large" and "small," but a configuration that allows setting three or more capacitor sizes may be adopted.

[0044] The column amplifier 124 is used to electrically amplify the signal read from the pixel unit 123. By amplifying the signal with the column amplifier 124, the signal level of the pixels is amplified relative to the noise generated by the subsequent column ADC 125, thereby effectively improving the signal-to-noise ratio. Furthermore, the gain of the column amplifier can be changed from the timing pulse control unit 121.

[0045] The image sensor 13 is equipped with two input memories in the column AMP 124, and by changing the gain of the column AMP 124, it is possible to output signals with two different gains for generating DGO composite images. As mentioned above, a DGO composite image is a composite image with an expanded dynamic range, generated by combining two images with different gains taken in a common single exposure using the DGO image sensor (in this case, image sensor 13). By having two input memories, two different gains can be applied separately to the signal read from the FD at a specific time and output. Therefore, although the amount of data increases, it is possible to obtain two images with different gains that are simultaneous. Note that the image sensor 13 may be configured to output three or more images with different gains that are simultaneous.

[0046] The column ADC 125 performs AD conversion on the read signals from the column AMP 124. The digitized signals are sequentially read out by the horizontal transfer circuit 126. The output of the horizontal transfer circuit 126 is input to the signal processing circuit 127.

[0047] The signal processing circuit 127 is a circuit that performs signal processing digitally. The signal processing circuit 127 can easily perform gain calculations by adding a fixed amount of offset value through digital processing of the signal, and by performing shift operations and multiplication. In addition, a pixel area that is intentionally shielded from light may be provided in the pixel section 123. In this case, the signal processing circuit 127 can perform a digital black level clamp operation using the shielded pixel area. The output of the signal processing circuit 127 is passed to the external output circuit 128.

[0048] The external output circuit 128 has a serializer function and converts the multi-bit input parallel signal from the signal processing circuit 127 into a serial signal. The external output circuit 128 also converts this serial signal into, for example, a Low Voltage Differential Signaling (LVDS) signal and outputs it to the outside of the image sensor 13.

[0049] Next, with reference to Figure 1C, the operation of the image sensor 13 during the generation of a DGO composite image will be described. Figure 1C is a conceptual diagram of the circuit with one column of the column AMP 124 extracted. As mentioned above, the image sensor 13 can change the gain of the column AMP 124 in order to generate a DGO composite image.

[0050] OP135 is an operational amplifier, and input capacitance and feedback capacitance are connected to it. C133 and C134 are input capacitances, and SW131 and SW132 are switches. The signal read from the pixel unit 123 is connected to C133 and C134 via SW131 and SW132.

[0051] C136 and C138 are feedback capacitors, and SW137 is a switch. SW137 can control the connection of C138. Since column AMP124 uses a capacitor, the amplification factor is (input capacitance / feedback capacitance).

[0052] As described above, column AMP124 has two input capacitors. First, by turning SW131 ON and SW132 and SW137 OFF, column AMP124 applies gains based on C133 and C136 to the input signal and outputs it to column ADC125. Next, by turning SW131 OFF and SW132 and SW137 ON, column AMP124 applies gains based on C134, C136, and C138 to the input signal and outputs it. This makes it possible to output two images with different gains.

[0053] Thus, the digital camera 100 is equipped with an image sensor 13 which is a DGO image sensor, and has the function of generating two images with different gains by taking a picture with a single exposure, and then generating a DGO composite image (second type image) by combining these two images. The digital camera 100 also has the function of generating a captured image (first type image) by taking a picture with a single exposure. In this case, the digital camera 100 controls the image sensor 13 to output a single image to which a single gain is applied. Furthermore, the digital camera 100 has the function of generating a multi-exposure composite image (third type image) by combining multiple captured images corresponding to multiple exposures. In this case, the digital camera 100 controls the image sensor 13 to output a single image to which a single gain is applied in each exposure. The image synthesis processing for generating the DGO composite image and the multi-exposure composite image is performed using the image processing unit 20.

[0054] The image sensor 13 may be configured to output signals with three or more different gains for a single exposure. In this case, the digital camera 100 can generate a composite image as a second type of image, based on three or more images with different gains captured in a single exposure. Even in this case, since the exposure involved in the composite image is only one common exposure, it is considered unlikely that a scene that could not be captured in a single exposure will be recorded in the composite image. Therefore, when describing DGO composite images below, the description also applies to composite images based on three or more images with different gains captured in a single exposure, as long as it is not technically contradictory.

[0055] Figure 2A shows an example of the configuration of a multi-exposure composite image file. The image file 200A recorded in this embodiment includes an area for recording metadata (Exif data 201) in accordance with the Exif standard, an area for recording compressed main image data 206, and an area for recording authenticity assurance information 223 (authenticity assurance information area 207).

[0056] For example, if the user instructs the image to be recorded in JPEG format, the thumbnail image data 205 and the main image data 206 will be recorded in JPEG format in the image file 200A. In addition, Exif data 201 will be recorded in the APP1 marker, etc., and the authenticity guarantee information area 207 will be recorded in the APP11 marker, etc.

[0057] Furthermore, if the user instructs the system to record in High Efficiency Image File Format (HEIF), the image file 200A will be recorded in HEIF format, and the Exif data 201 and authenticity information area 207 will be recorded in a MetaDataBox or similar location. Similarly, if the user instructs the system to record in RAW format, the Exif data 201 and authenticity information area 207 will also be recorded in a designated area such as a MetaDataBox.

[0058] Image file 200A may be recorded in formats other than those described above.

[0059] Exif data 201 may also record information 202 indicating the presence or absence of authenticity assurance information and a link 203 to the authenticity assurance information. Furthermore, the MakerNote 204 included in Exif data 201 may contain manufacturer-specific metadata generated using the metadata generation and analysis unit 70, which is generally kept confidential.

[0060] The authenticity assurance information area 207 includes authenticity assurance information 223 (first authenticity assurance information) which contains information to guarantee the authenticity of the image data 206. The authenticity assurance information 223 includes the action, Actor (name of the camera or software that generated the image), various hash values, thumbnail image data 224, and signature.

[0061] The action indicates (identifies) whether the image data 206 is a multi-exposure composite image, a non-composite image (an image captured in a single exposure), or a DGO composite image. If the image data 206 is a multi-exposure composite image, the action has the value "Edit" (first information) as shown in Figure 2A. If the image data 206 is either a non-composite image or a DGO composite image, the action has the value "Generate" (second information).

[0062] Thumbnail image data 224 is thumbnail image data corresponding to the main image data 206 at the time when authenticity assurance information 223 is added to the image file 200A.

[0063] The various hash values ​​include the hash value of the main image data 206, the hash value of the thumbnail image data 224, and the hash value of the metadata (Exif data 201 in the example in Figure 2A).

[0064] Thus, the authenticity assurance information 223 includes the hash values ​​of both the main image data 206 and the thumbnail image data 224, and these hash values ​​are signed. Therefore, the authenticity assurance information 223 can be used to guarantee the authenticity of the main image data 206 and the thumbnail image data 224 (if the authenticity is compromised, the signature verification will fail, so if the signature verification is successful, it is guaranteed that the main image data 206 and the thumbnail image data 224 are authentic).

[0065] Figure 2B shows another example of the configuration of a multi-exposure composite image file. In the example in Figure 2A, the authenticity assurance information area 207 contains only one authenticity assurance information 223 (first authenticity assurance information), but as shown in Figure 2B, the authenticity assurance information area 207 may contain multiple authenticity assurance information (multiple second authenticity assurance information). In the example in Figure 2B, the authenticity assurance information area 207 contains three authenticity assurance information 217, 219, and 221 corresponding to the three source images used to generate the multi-exposure composite image, in addition to the authenticity assurance information 223 for guaranteeing the authenticity of the main image data 206, which is a multi-exposure composite image. The thumbnail image data 218, 220, and 222 stored in the authenticity assurance information 217, 219, and 221 are thumbnail image data corresponding to each source image at the time the authenticity assurance information was added to each captured source image. The thumbnail image data 224 stored in the authenticity assurance information 223 is, as in Figure 2A, a thumbnail image data corresponding to a multi-exposure composite image (main image data 206). Since the source image is a non-composite image, the action stored in the authenticity assurance information 217, 219, and 221 has the value "Generate".

[0066] In the example in Figure 2B, the latest authenticity assurance information for the multi-exposure composite image is the fourth (bottommost) authenticity assurance information, 223. When the system control unit 50 records multiple authenticity assurance information in the authenticity assurance information area 207, it may store management information at the beginning of the authenticity assurance information area 207, such as a link to each authenticity assurance information, the number of authenticity assurance information items, and information indicating which is the latest authenticity assurance information. Alternatively, the system control unit 50 may record multiple authenticity assurance information sequentially without recording management information in the authenticity assurance information area 207, and treat the last recorded authenticity assurance information as the latest authenticity assurance information. Alternatively, the system control unit 50 may store information in the authenticity assurance information area 207 indicating which authenticity assurance information was added when the non-composite image was captured.

[0067] Authenticity assurance information 217 includes information to guarantee the authenticity of the source image and the thumbnail image (thumbnail image data 218) corresponding to the source image. The structure of authenticity assurance information 217 is the same as that of authenticity assurance information 223. However, the hash value of the main image data included in authenticity assurance information 217 is not the hash value of the main image data 206 of image file 200B, but the hash value of the source image data (main image data 206 in Figure 2C, described later) used to generate the main image data 206. Also, the hash value of the thumbnail image data included in authenticity assurance information 217 is the hash value of the thumbnail image data 218 corresponding to the source image data. The structure of authenticity assurance information 219 and 221 is the same as that of authenticity assurance information 217.

[0068] Since the image file 200B contains authenticity assurance information 217, 219, and 221, the digital camera 100 can verify the authentic origin of the main image data 206, which is a multi-exposure composite image. For example, the digital camera 100 can verify the authenticity of the thumbnail image data 218 corresponding to the source image data by verifying the signature of the hash value of the thumbnail image data in the authenticity assurance information 217.

[0069] Figure 2C shows an example of the structure of a non-composite image file. As shown in Figure 2C, the structure of image file 200C may be the same as the structure of image file 200A in Figure 2A. In the example in Figure 2C, it is assumed that image file 200C is an image file that stores the source image corresponding to the authenticity assurance information 217 in Figure 2B as the main image.

[0070] Figure 2D shows an example of the structure of a DGO composite image file. As shown in Figure 2D, the structure of image file 200D may be the same as the structure of image file 200A in Figure 2A. In the example in Figure 2D, the authenticity assurance information area 207 contains authenticity assurance information 208 to guarantee the authenticity of the main image data 206, which is a DGO composite image. The thumbnail image data 209 stored in the authenticity assurance information 208 is the thumbnail image data corresponding to the main image data 206, which is a DGO composite image. The number of exposures involved in a DGO composite image is only one, and it is unlikely that a scene that could not be captured in an image taken with a single exposure will be recorded in the DGO composite image. Therefore, the action stored in the authenticity assurance information 208 has the value "Generate," just as in the case of a non-composite image.

[0071] Figure 3 is a flowchart of the process for adding authenticity assurance information to an image. Here, we will explain the case where a DGO composite image is generated, and the case where a multi-exposure composite image is generated by multiplexing three exposure images (non-composite images). Unless otherwise specified, the processing of each step in this flowchart is realized by the system control unit 50 executing a program stored in the non-volatile memory 51. In this embodiment, the number of source images used to generate a multi-exposure composite image is not limited to three. Also, the method of combining multiple source images (exposure images) is not limited to multiplexing, but may be other combining methods (for example, multi-shot HDR combining, multi-shot NR combining, multi-shot panorama combining, etc.).

[0072] The user can display the HDR shooting menu on the display unit 23 by operating the control unit 63. The user can then set the HDR shooting menu to generate a DGO composite image using DGO compositing (a process that combines two images with different gains captured in a single exposure). Subsequently, when the user gives a single shooting (exposure) command to the digital camera 100, the digital camera 100 takes a single exposure. As a result of the single exposure, two images with different gains are generated, and the digital camera 100 generates a DGO composite image by combining these two images. Each time a DGO composite image is generated in the digital camera 100, the processing in this flowchart is performed on the generated DGO composite image. The two images with different gains used to generate the DGO composite image are, for example, a properly exposed image and an underexposed image, but are not limited to these. Also, in the above description, the DGO composite image is assumed to be a composite image with an expanded dynamic range, but it may also be a composite image with reduced noise. In this case, the digital camera 100 controls the image sensor 13 to generate both a properly exposed image and an overexposed image by taking a picture with a single exposure. The digital camera 100 then generates a composite image with reduced noise in the dark areas by darkening the overexposed image by the number of stops of difference between it and the properly exposed image, and then compositing this image into the dark areas of the properly exposed image.

[0073] Furthermore, the user can operate the control unit 63 to display the multiple image composition setting menu on the display unit 23, and set the number of images (source images) to be used for composition to three on the multiple image composition setting menu. After that, when the user takes three shots (exposures) with the digital camera 100, three source images (non-composite images) are generated, and a multi-exposure composite image based on the three source images is generated. Each time an image (non-composite image or multi-exposure composite image) is generated by the digital camera 100, the processing of this flowchart is performed on the generated image.

[0074] In S301, the system control unit 50 determines whether the setting for adding authenticity assurance information is ON. If the setting for adding authenticity assurance information is ON, the process proceeds to S302; otherwise, the process proceeds to S305.

[0075] It should be noted that the user has previously configured the authenticity guarantee information setting. For example, the user can display the settings menu on the display unit 23 by operating the control unit 63, and on the settings menu, they can select ON (enable) or OFF (disable) as the additional setting.

[0076] In S302, the system control unit 50 determines whether the image to be processed (the first image) is either a non-composite image or a DGO composite image. If the image to be processed is either a non-composite image or a DGO composite image, the process proceeds to S303; otherwise (if the image to be processed is a multi-exposure composite image), the process proceeds to S304.

[0077] In S303, the system control unit 50 generates authenticity assurance information for the image to be processed (here, a non-composite image or a DGO composite image) using the hash value generation unit 72 and the signature generation / verification unit 73, and attaches (associates) the generated authenticity assurance information to the image to be processed. In this case, an image file such as the image file 200C shown in Figure 2C or the image file 200D shown in Figure 2D is generated. Since the image to be processed is a non-composite image or a DGO composite image, the value "Generate" is set for the action stored in the authenticity assurance information 217 or authenticity assurance information 208, as shown in Figure 2C or Figure 2D. In addition, the hash value generation unit 72 calculates the hash value of the main image data 206, the hash value of the thumbnail image data 218 or 209, and the hash value (not shown) of the metadata (Exif data 201), and these hash values ​​are stored in the authenticity assurance information 217 or 208.

[0078] In the case of multiple image synthesis, it is assumed that three source images will be captured, so the process in S303 will be executed three times, and three source image files will be generated. For example, the second source image file has the file structure shown in Figure 2C, and its authenticity assurance information is the authenticity assurance information 219 shown in Figure 2B. Similarly, the third source image file has the file structure shown in Figure 2C, and its authenticity assurance information is the authenticity assurance information 221 shown in Figure 2B.

[0079] In S304, the system control unit 50 generates authenticity assurance information for the image to be processed (in this case, a multi-exposure composite image) using the hash value generation unit 72 and the signature generation / verification unit 73, and attaches (associates) the generated authenticity assurance information to the image to be processed. In this case, an image file like the image file 200A shown in Figure 2A is generated. Since the image to be processed is a multi-exposure composite image, the action stored in the authenticity assurance information 223 is set to the value "edit," as shown in Figure 2A. In addition, the hash value generation unit 72 calculates the hash value of the main image data 206, the hash value of the thumbnail image data 224, and the hash value (not shown) of the metadata (Exif data 201), and these hash values ​​are stored in the authenticity assurance information 223.

[0080] In S305, the system control unit 50 refrains from adding authenticity assurance information to the image to be processed.

[0081] In S306, the system control unit 50 saves an image file containing an image with authenticity assurance information attached (if S303 or S304 is passed) or an image without authenticity assurance information attached (if S305 is passed) to the external recording medium 91.

[0082] In the above explanation, it was assumed that the action in authenticity assurance information 223 has the value "edit" (first information), and the action in authenticity assurance information 217, 219, 221, and 208 has the value "generate" (second information). Authenticity assurance information 217, 219, 221, 223, and 208 identify whether the image whose authenticity should be guaranteed is either a non-composite image, a DGO composite image, or a multi-exposure composite image, based on the value of the action. However, the method of configuring authenticity assurance information to identify whether an image is either a non-composite image, a DGO composite image, or a multi-exposure composite image is not limited to using the value of the action ("edit" or "generate"). For example, the action in authenticity assurance information 217, 219, 221, and 208 may have the value "generate," while authenticity assurance information 223 may be configured not to include an action. In this case, the absence of an action in the authenticity assurance information means that the image is a multi-exposure composite image.

[0083] Furthermore, in the above explanation, when the image to be processed is a multi-exposure composite image, authenticity assurance information with an "edit" action (for example, authenticity assurance information 223 in Figure 2A) is associated with the multi-exposure composite image (S304). However, when the image to be processed is a multi-exposure composite image, the digital camera 100 may be configured to refrain from associating the authenticity assurance information with the image to be processed (multi-exposure composite image). In this case, it is possible to determine whether the image is a non-composite image or a DGO composite image based on whether or not the authenticity assurance information is associated with the image.

[0084] As described above, according to the first embodiment, when the first image (for example, the main image data 206 in Figure 2A, Figure 2C, or Figure 2D) is either a first type image (for example, the main image data 206 in Figure 2C) or a second type image (for example, the main image data 206 in Figure 2D), the digital camera 100 associates the first image with first authenticity assurance information (for example, authenticity assurance information 217 in Figure 2C, or authenticity assurance information 208 in Figure 2D), which includes information to guarantee the authenticity of the first image. Here, the first type image is a non-composite image (captured image) taken in a single exposure. The second type image is a composite image (single exposure composite image) generated by combining multiple non-composite images (captured images) taken in a common single exposure.

[0085] A single-exposure composite image is, for example, the DGO composite image described above, but is not limited to this. In composite images generated by combining multiple non-composite images taken in a common single exposure, the possibility of the phenomenon where "scenes that could not be captured in a single exposure are recorded in the composite image" is low. Therefore, any composite image generated by combining multiple non-composite images taken in a common single exposure is included in the single-exposure composite image of this embodiment.

[0086] Therefore, according to this embodiment, it is possible to associate appropriate authenticity assurance information with single-exposure composite images, which are unlikely to contain scenes that would not be possible in an image captured in a single exposure.

[0087] [Second Embodiment] In the first embodiment, as described in the explanation of S304 in Figure 3, an image file having the structure shown in Figure 2A is generated. In other words, in the first embodiment, only one authenticity assurance information 223 corresponding to the main image data 206 of the multi-exposure composite image is added to the multi-exposure composite image. In contrast, the second embodiment describes a case in which authenticity assurance information corresponding to the source image is also recorded as origin for the multi-exposure composite image.

[0088] In the second embodiment, the basic configuration of the digital camera 100 is the same as in the first embodiment. The following will mainly describe the differences from the first embodiment.

[0089] In the second embodiment, at S304 in Figure 3, the system control unit 50 acquires multiple authenticity assurance pieces of information (multiple second authenticity assurance pieces of information) corresponding to multiple captured images (non-composite images) used as source images. Each of the multiple authenticity assurance pieces of information acquired here is associated with one of the multiple captured images. Here, the number of source images (captured images) is assumed to be three. The system control unit 50 adds the three authenticity assurance pieces of information corresponding to the three captured images (non-composite images) as authenticity assurance pieces of information 217, 219, and 221 (multiple second authenticity assurance pieces of information) to the multi-exposure composite image. Since the source images are non-composite images, the action stored in the authenticity assurance pieces of information 217, 219, and 221 has the value "generate". The system control unit 50 also adds authenticity assurance piece of information 223 (first authenticity assurance piece of information) corresponding to the main image data 206 of the multi-exposure composite image to the multi-exposure composite image.

[0090] Thus, according to the second embodiment, it is possible to add authenticity assurance information 223 to the multi-exposure composite image, which includes the history of what source images were used to generate the multi-exposure composite image (authenticity assurance information 217, 219, 221), and which has an action with the value "edit".

[0091] [Third Embodiment] In the first and second embodiments, three source images are captured when generating a multi-exposure composite image. However, the source images may be selected from pre-captured image files. In the third embodiment, a case is described in which a multi-exposure composite image is generated by using images from a pre-captured image file as source images.

[0092] In the third embodiment, the basic configuration of the digital camera 100 is the same as in the first embodiment. The following will mainly describe the differences from the first embodiment.

[0093] The external recording medium 91 is assumed to contain two or more image files, each containing a captured image (non-composite image). The user can select multiple image files (multiple source image files) to be used to generate a multi-exposure composite image from the two or more image files stored on the external recording medium 91 by operating the operation unit 63. Here, it is assumed that three image files are selected. The system control unit 50 controls the image processing unit 20 to generate a multi-exposure composite image from the captured images in the three image files. Once the multi-exposure composite image is generated, the flowchart in Figure 3 is performed on the multi-exposure composite image.

[0094] In the third embodiment, at S304 in Figure 3, the system control unit 50 determines whether or not authenticity assurance information has been added to the source image file.

[0095] If the source image file has authenticity assurance information attached to it to guarantee the authenticity of the source image, the system control unit 50 inherits the authenticity assurance information from the source image file and also attaches the authenticity assurance information to the multi-exposure composite image. As a result, an image file like the image file 200B shown in Figure 2B is generated. Alternatively, the system control unit 50 may not attach the authenticity assurance information from the source image file to the multi-exposure composite image, but instead attach only the authenticity assurance information corresponding to the main image data of the multi-exposure composite image to the multi-exposure composite image. In this case, an image file like the image file 200A shown in Figure 2A is generated.

[0096] If authenticity assurance information is not attached to the source image file, the system control unit 50 refrains from attaching authenticity assurance information to the multi-exposure composite image.

[0097] Thus, according to the third embodiment, authenticity assurance information for guaranteeing the authenticity of the multiple exposure composite image is added to the multiple exposure composite image only when authenticity assurance information for guaranteeing the authenticity of the source image (non-composite image) is added to the source image. Therefore, according to this embodiment, it is possible to correctly record the origin information of the authenticity assurance information of the multiple exposure composite image.

[0098] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0099] [summary] The embodiments described above disclose, but are not limited to, the inventions shown in at least the following items. (Item 1) When the first image is either a first type image or a second type image, the system includes association means for associating the first image with first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image. The aforementioned first type of image is a non-composite image captured in a single exposure. The second type of image described above is a composite image generated by combining multiple non-composite images taken in a single common exposure. An image processing apparatus characterized by the following: (Item 2) If the first image is a third type image, the association means associates the first authenticity information with the first image. The third type of image is a composite image generated by combining multiple captured images corresponding to multiple exposures. The first authenticity information is configured to identify whether the first image is either a first type image or a second type image, or whether the first image is a third type image. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 3) If the first image is of the third type, the first authenticity assurance information includes first information indicating that the first image is of the third type. If the first image is either a first-type image or a second-type image, the first authenticity assurance information includes second information indicating that the first image is either a first-type image or a second-type image. The image processing apparatus according to item 2, characterized in that (Item 4) If the first image is of the third type, the association means associates a plurality of second authenticity assurance pieces of information with the first image. The plurality of second authenticity assurance information includes a plurality of thumbnail images corresponding to the plurality of captured images, and information for guaranteeing the authenticity of the plurality of thumbnail images. The image processing apparatus according to item 2 or 3, characterized in that it is an image processing apparatus. (Item 5) The aforementioned plurality of second authenticity assurance information includes information for ensuring the authenticity of the plurality of captured images, The image processing apparatus according to item 4, characterized in that (Item 6) The system further comprises a synthesis means for generating the first image, which is the third type of image, by combining the aforementioned plurality of captured images, If each of the multiple second authenticity information is associated with a separate one of the multiple captured images, and each of the multiple second authenticity information includes information for guaranteeing the authenticity of the corresponding captured image, the association means associates the first authenticity information with the first image which is the third type of image. If each of the plurality of second authenticity assurance pieces is not associated with a separate one of the plurality of captured images, the association means refrains from associating the first authenticity assurance piece with the first image which is the third type of image. The image processing apparatus according to item 2, characterized in that (Item 7) If each of the plurality of second authenticity information is associated with a separate one of the plurality of captured images, the association means associates the plurality of second authenticity information with the first image which is the third type image. The plurality of second authenticity assurance information includes a plurality of thumbnail images corresponding to the plurality of captured images, and information for guaranteeing the authenticity of the plurality of thumbnail images. The image processing apparatus according to item 6, characterized in that (Item 8) The system further includes a selection means that, in accordance with user instructions, selects multiple image files from a recording medium that stores two or more image files, each containing a captured image. The synthesis means generates the first image, which is the third type of image, by synthesizing the multiple captured images contained in the multiple image files. The image processing apparatus according to item 6 or 7, characterized in that it is an image processing apparatus. (Item 9) If the first image is a third type image, the association means refrains from associating the first authenticity information with the first image. The third type of image is a composite image generated by combining multiple captured images corresponding to multiple exposures. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 10) The aforementioned multiple non-composite images captured in a single common exposure are multiple images captured using an image sensor configured to generate multiple images with different gains applied in a single common exposure. An image processing apparatus according to any one of items 1 to 9, characterized in that (Item 11) An image processing apparatus described in any one of items 1 to 10, A means for capturing a non-composite image taken with a single exposure, and a non-composite image taken without exposure, An imaging device characterized by comprising: (Item 12) A control method performed by an image processing device, If the first image is either a first-type image or a second-type image, the system includes an association step of associating the first image with first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image. The aforementioned first type of image is a non-composite image captured in a single exposure. The second type of image described above is a composite image generated by combining multiple non-composite images taken in a single common exposure. A control method characterized by the following: (Item 13) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 10. The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0100] 13…Image sensor, 20…Image processing unit, 25…Memory, 50…System control unit, 51…Non-volatile memory, 63…Operation unit, 72…Hash value generation unit, 73…Signature generation / verification unit

Claims

1. When the first image is either a first type image or a second type image, the system includes association means for associating the first image with first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image. The first type of image described above is a non-composite image taken in a single exposure. The second type of image is a composite image generated by combining multiple non-composite images taken in a single common exposure. An image processing apparatus characterized by the following:

2. If the first image is a third type image, the association means associates the first authenticity assurance information with the first image. The third type of image is a composite image generated by combining multiple captured images corresponding to multiple exposures. The first authenticity information is configured to identify whether the first image is either a first type image or a second type image, or whether the first image is a third type image. The image processing apparatus according to feature 1.

3. If the first image is of the third type, the first authenticity assurance information includes first information indicating that the first image is of the third type. If the first image is either a first type image or a second type image, the first authenticity information includes second information indicating that the first image is either a first type image or a second type image. The image processing apparatus according to claim 2.

4. If the first image is of the third type, the association means associates a plurality of second authenticity assurance pieces of information with the first image. The plurality of second authenticity assurance information includes a plurality of thumbnail images corresponding to the plurality of captured images, and information for guaranteeing the authenticity of the plurality of thumbnail images. The image processing apparatus according to claim 2.

5. The aforementioned plurality of second authenticity assurance information includes information for ensuring the authenticity of the plurality of captured images, The image processing apparatus according to feature 4.

6. The system further comprises a synthesis means for generating the first image, which is the third type of image, by combining the aforementioned plurality of captured images, When each of the multiple second authenticity assurance pieces is associated with a separate one of the multiple captured images, and each of the multiple second authenticity assurance pieces includes information for ensuring the authenticity of the corresponding captured image, the association means associates the first authenticity assurance piece with the first image which is the third type of image. If each of the plurality of second authenticity assurance pieces is not associated with a separate one of the plurality of captured images, the association means refrains from associating the first authenticity assurance piece with the first image which is the third type of image. The image processing apparatus according to claim 2.

7. If each of the plurality of second authenticity information is associated with a separate one of the plurality of captured images, the association means associates the plurality of second authenticity information with the first image which is the third type image. The plurality of second authenticity assurance information includes a plurality of thumbnail images corresponding to the plurality of captured images, and information for guaranteeing the authenticity of the plurality of thumbnail images. The image processing apparatus according to claim 6.

8. The system further includes a selection means that, in accordance with user instructions, selects multiple image files from a recording medium that stores two or more image files, each containing a captured image. The synthesis means generates the first image, which is the third type of image, by synthesizing the multiple captured images contained in the multiple image files. The image processing apparatus according to claim 6.

9. If the first image is a third type image, the association means refrains from associating the first authenticity information with the first image. The third type of image is a composite image generated by combining multiple captured images corresponding to multiple exposures. The image processing apparatus according to feature 1.

10. The aforementioned multiple non-composite images captured in a single common exposure are multiple images captured using an image sensor configured to generate multiple images with different gains applied in a single common exposure. The image processing apparatus according to feature 1.

11. An image processing apparatus according to any one of claims 1 to 10, A means for capturing a non-composite image taken with a single exposure, and a non-composite image taken without exposure, An imaging device characterized by comprising:

12. A control method performed by an image processing device, If the first image is either a first-type image or a second-type image, the system includes an association step of associating the first image with first authenticity assurance information, which includes information for guaranteeing the authenticity of the first image. The first type of image described above is a non-composite image taken in a single exposure. The second type of image is a composite image generated by combining multiple non-composite images taken in a single common exposure. A control method characterized by the following:

13. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 10.

Images