Image processing apparatus and method

Abstract

The present disclosure relates to an image processing device and method that allow the authenticity of an image to be more precisely determined. An optical image from a subject is captured, thereby acquiring an image of the subject. 3D information is acquired from the optical image with the same optical axis as the subject image, and a signature of the subject image and the 3D information is generated. Also, an image is compared with 3D information acquired with the same optical axis as the image, to confirm the authenticity of the image. The present disclosure can be applied, for example, to an image processing device or an image processing method.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and method, and particularly to an image processing apparatus and method that can more accurately determine the authenticity of an image.

Background Art

[0002] Conventionally, a method has been proposed in which an imaging image or the like is converted into a hash value in a digital camera or the like, and an electronic signature using the hash value is attached to the imaging image for use in detecting forgery of the imaging image. However, with this method, it has not been possible to detect a fake image generated by so-called trick photography or the like.

[0003] Therefore, a method of detecting a fake image based on the consistency between information indicating the focal length at the time of imaging and the focal length obtained from the imaging image has been considered (see, for example, Patent Document 1). Further, a method of detecting a fake image by determining whether the subject of the imaging image is flat based on the multi-point distance measurement data of the camera has been considered (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the method described in Patent Document 1 only determines the consistency of the focal length and cannot confirm the consistency of the surface features of the subject. Similarly, the method described in Patent Document 2 only determines whether the subject is flat or not and cannot confirm the consistency of the surface features of the subject. Furthermore, there was no guarantee that the subject in the captured image and the subject (the object being measured) in the distance measurement data were the same. As a result, there was a risk that false image detection would be inaccurate.

[0006] This disclosure is made in light of these circumstances and aims to enable a more accurate determination of the authenticity of images. [Means for solving the problem]

[0007] One aspect of this technology is an image processing device comprising: an image acquisition unit that acquires an image of a subject by capturing an optical image from the subject; and a 3D information acquisition unit that acquires 3D information showing the surface irregularities of the subject from the optical image on the same optical axis as the image. A plane determination unit determines whether the shape of the subject is planar or not based on the 3D information, and if the plane determination unit determines that the shape of the subject is not planar, The image processing apparatus comprises a signature generation unit that generates signatures for the aforementioned image and the aforementioned 3D information.

[0008] One aspect of this technology is an image processing method which acquires an image of a subject by capturing an optical image from the subject, and acquires 3D information showing the surface irregularities of the subject from the optical image on the same optical axis as the image. Based on the 3D information, it is determined whether the shape of the subject is planar or not, and if it is determined that the shape of the subject is not planar, This is an image processing method for generating signatures for the aforementioned image and the aforementioned 3D information.

[0009] Image processing equipment for other aspects of this technology is: An optical axis determination unit that determines whether 3D information showing the surface irregularities of the subject was acquired on the same optical axis as the image of the subject, and Images and The aforementioned A signature verification unit confirms the validity of the signature of information including 3D information, and if the validity of the signature is confirmed, it confirms the authenticity of the image by comparing the surface irregularities of the subject estimated from the image with the surface irregularities of the subject shown by the 3D information. Furthermore, if the 3D information is not acquired on the same optical axis as the image, the image is deemed to lack authenticity. This is an image processing apparatus equipped with an image verification processing unit.

[0010] Other aspects of this technology, specifically image processing methods, It is determined whether the 3D information showing the surface irregularities of the subject was acquired on the same optical axis as the image of the subject, Images and The aforementioned The validity of the signature of the information including 3D information is verified, and if the validity of the signature is verified, the authenticity of the image is verified by comparing the surface topography of the subject estimated from the image with the surface topography of the subject shown by the 3D information. Furthermore, if the 3D information is not acquired on the same optical axis as the image, the image is deemed to lack authenticity. This is an image processing method.

[0011] In one aspect of this technology, an image processing apparatus and method are used to acquire an image of a subject by capturing an optical image of the subject, and 3D information indicating the topography of the subject is obtained from the optical image on the same optical axis as the image. Based on that 3D information, it is determined whether the shape of the subject is flat or not, and if it is determined that the shape of the subject is not flat, A signature is generated for the image and 3D information.

[0012] In other aspects of this technology, the image processing apparatus and method, It is determined whether the 3D information showing the surface texture of the subject was acquired on the same optical axis as the image of that subject, and the image and 3D information are then compared. The validity of the signature of the information containing the above is verified, and if the validity of the signature is verified, the authenticity of the image is confirmed by comparing the surface topography of the subject estimated from the image with the surface topography of the subject shown by the 3D information. Furthermore, if the 3D information is not acquired on the same optical axis as the image, the image is deemed to lack authenticity. It will be done. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing examples of the main configurations of an image processing system. [Figure 2] This block diagram shows examples of the main components of an imaging device. [Figure 3] This figure shows an example of the main configuration of the sensor unit. [Figure 4] This is a block of data showing the main server configuration examples. [Figure 5] This diagram shows an example of a device public key database. [Figure 6] This block diagram shows examples of the main configurations of terminal devices. [Figure 7] This flowchart shows an example of the key processing flow. [Figure 8] This is a diagram illustrating an example of a displayed image. [Figure 9]A diagram for explaining an example of a display image. [Figure 10] A diagram for explaining an example of information stored in a memory unit. [Figure 11] A diagram for explaining an example of a display image. [Figure 12] A diagram for explaining an example of a display image. [Figure 13] A diagram for explaining an example of a display image. [Figure 14] A flowchart showing an example of the flow of preview reliability determination processing. [Figure 15] A diagram for explaining an example of a preview image and 3D information. [Figure 16] A flowchart showing an example of the flow of imaging processing. [Figure 17] A timing chart for explaining 3D information generation timing. [Figure 18] A diagram for explaining an example of a captured image and 3D information. [Figure 19] A diagram for explaining a main configuration example of an image file. [Figure 20] A diagram for explaining an example of a display image. [Figure 21] A flowchart showing an example of the flow of imaging processing. [Figure 22] A flowchart showing an example of the flow of imaging processing. [Figure 23] A flowchart showing an example of the flow of imaging processing. [Figure 24] A flowchart showing an example of the flow of imaging processing. [Figure 25] A flowchart showing an example of the flow of imaging processing. [Figure 26] A flowchart showing an example of the flow of imaging processing. [Figure 27] A flowchart showing an example of the flow of confirmation processing. [Figure 28] A flowchart showing an example of the flow of image confirmation processing. [Figure 29]This flowchart shows an example of the process for displaying comparison results. [Figure 30] This is a diagram illustrating an example of a displayed image. [Figure 31] This flowchart shows an example of the image verification process. [Figure 32] This flowchart shows an example of the process for displaying comparison results. [Figure 33] This is a diagram illustrating an example of a displayed image. [Figure 34] This flowchart shows an example of the image verification process. [Figure 35] This flowchart shows an example of the process for displaying comparison results. [Figure 36] This flowchart shows an example of the confirmation process flow. [Figure 37] This flowchart shows an example of the confirmation process flow. [Figure 38] This flowchart shows an example of the confirmation process flow. [Figure 39] This flowchart shows an example of the confirmation process flow. [Figure 40] This flowchart shows an example of the confirmation process flow. [Figure 41] This flowchart shows an example of the confirmation process flow. [Figure 42] This flowchart shows an example of the confirmation process flow. [Figure 43] This diagram illustrates the main structure of an image file. [Figure 44] This flowchart shows an example of the image processing flow. [Figure 45] This flowchart shows an example of the confirmation process flow. [Figure 46] This flowchart shows an example of the image processing flow. [Figure 47] This diagram illustrates an example of processed images and 3D information. [Figure 48] This is a block diagram showing common computer configurations. [Modes for carrying out the invention]

[0014] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order. 1. Detection of fake images 2. Detection of fake images using 3D information 3. Miniature signature 4. Reflection of image processing results in 3D information. 5. Application Examples 6. Addendum

[0015] <1. Detection of fake images> Conventionally, methods have been proposed to detect tampering with captured images by converting captured images into hash values ​​and attaching an electronic signature using those hash values ​​to the captured images. However, this method could not detect fake images generated by so-called trick photography. A fake image is a captured image that makes a non-existent situation appear real; that is, a captured image of a non-existent situation that is made to look like an image obtained by capturing a real situation. Trick photography is a photographic technique that generates fake images through the use of tools or photographic ingenuity.

[0016] Therefore, a method for detecting false images has been devised, for example, as described in Patent Document 1, based on the consistency between information indicating the focal length at the time of imaging and the focal length obtained from the captured image. Alternatively, a method for detecting false images has been devised, for example, as described in Patent Document 2, by determining whether or not the subject in the captured image is flat based on multi-point distance measurement data from the camera.

[0017] However, these methods risked inaccurate detection of false images. For example, the method described in Patent Document 1 only determined the consistency of the focal length and could not confirm the consistency of the subject's surface features, etc. Therefore, even if a false image was generated by, for example, taking a photograph of a face, and was designed to appear as if it was a photograph of the subject in that photograph, if the focal length matched the metadata, the method described in Patent Document 1 would have difficulty detecting that false image.

[0018] Furthermore, the method described in Patent Document 2 had the potential to obtain distance measurement data on a different optical axis than the captured image. Therefore, trick photography was easily performed, for example, by capturing a facial photograph and detecting the distance to a person other than the subject of the facial photograph. Consequently, if distance measurement data with irregularities was obtained through such trick photography, it was difficult to detect the false image using the method described in Patent Document 2.

[0019] <2. Detection of false images using 3D information> Therefore, in an imaging device that captures a subject and generates an image, 3D information is acquired on the same optical axis as the image, and a signature is generated for both the image and the 3D information.

[0020] For example, an image processing device may include an image acquisition unit that acquires an image of a subject by capturing an optical image of the subject, a 3D information acquisition unit that acquires 3D information from the optical image on the same optical axis as the image, and a signature generation unit that generates signatures for the image and the 3D information.

[0021] For example, in an image processing method, an image of a subject is obtained by capturing an optical image of the subject, 3D information is obtained from that optical image on the same optical axis as the image, and a signature is generated for both the image and the 3D information.

[0022] Here, the image generated by the image generation unit is an image obtained by capturing an optical image from the subject, that is, an image obtained by the image generation unit receiving light from the subject and performing photoelectric conversion. This image may be a RAW image or a YUV image. Furthermore, this image may be encoded as, for example, a JPEG (Joint Photographic Experts Group) image. Furthermore, this image may be a still image or a moving image.

[0023] Furthermore, the 3D information (three-dimensional information) generated by the 3D information generation unit may be distance-related information of the subject. Also, the 3D information may be information generated using that distance-related information. Also, the 3D information may include both. That is, "acquisition" of 3D information may be detecting this distance-related information from an optical image. Also, "acquisition" of 3D information may be detecting distance-related information from an optical image and generating other information from the detected distance-related information.

[0024] This distance-related information may be the distance information itself from the image processing device to the subject, such as a depth map. Alternatively, this distance-related information may be information necessary to calculate the distance information from the image processing device to the subject, such as phase difference data or ToF (Time of Flight) data. Furthermore, this distance-related information may be a collection of disparity images (for example, a collection of images taken by swinging the imaging device from side to side in 3D swing panorama mode). For example, as described in Japanese Patent Application Publication No. 2012-70154, disparity images can be generated by performing swing shooting, which involves swinging the imaging device from side to side.

[0025] Furthermore, in an optical system, the optical axis refers to the principal ray that passes through the center of the light beam that passes through the entire system. "Acquiring 3D information from the optical image on the same optical axis as the image" means that the optical axis of the optical image from which the image is obtained and the optical axis of the optical image from which the 3D information is obtained are the same. In other words, in this case, the image processing device acquires both the image and 3D information from a single optical image, for example. Alternatively, the image processing device may split that single optical image into two optical images (identical optical images) using a beam splitter (half mirror) with a prism or the like, acquire the image from one optical image, and acquire the 3D information from the other optical image.

[0026] By having the same optical axis for the optical image used to obtain the image and the same optical axis for the optical image used to obtain the 3D information, it is possible to obtain 3D information from the same angle as the image for subjects within the field of view (angle of view) of the image. In other words, "obtaining 3D information from the optical image using the same optical axis as the image" can also be said to mean obtaining 3D information from the same angle as the image for subjects within the field of view of the image. The range of the 3D information obtained from the optical image is arbitrary. For example, the range of the 3D information may be the same as the field of view of the image, or it may include part or all of that field of view. For example, 3D information from the same angle as the image may be obtained for multiple locations within the field of view of the image.

[0027] An image processing device that determines the authenticity of an image can detect tampering with the image and 3D information based on such a signature. In other words, the image processing device can determine the authenticity of an image using the untampered image and 3D information. Furthermore, since distance-related information is detected on the same optical axis as the image, it is difficult to perform trick photography, such as taking a facial photograph and then detecting the distance to a person other than the subject of the facial photograph. That is, it is difficult to obtain 3D information indicating that the subject has bumps and depressions as 3D information corresponding to a false image obtained by taking a facial photograph. Therefore, the image processing device that determines the authenticity of an image can detect (identify) false images more accurately. In other words, the image processing device can determine the authenticity of an image more accurately.

[0028] Furthermore, in an imaging device for determining the authenticity of an image, the authenticity of the image is determined using 3D information obtained on the same optical axis as the image.

[0029] For example, an image processing device may be equipped with an image verification processing unit that verifies the authenticity of an image by comparing the image with 3D information acquired on the same optical axis as the image.

[0030] For example, in image processing methods, the authenticity of an image is verified by comparing it with 3D information acquired on the same optical axis as the image.

[0031] By comparing distance-related information (3D information) at multiple points within an image with the image itself, the image processing device can compare the surface topography of the subject. In other words, the image processing device can detect (identify) a false image based on whether the surface topography of the subject matches between the image and the 3D information. Therefore, the image processing device can detect (identify) false images more accurately than simply determining whether the focal length matches.

[0032] Furthermore, as mentioned above, because distance-related information is detected on the same optical axis as the image, trick photography becomes difficult. For example, it becomes difficult to obtain 3D information indicating that the subject has uneven surfaces as 3D information corresponding to a fake image obtained by capturing a facial photograph. Therefore, the image processing device can detect (identify) fake images more accurately.

[0033] In other words, by doing this, the image processing device can determine the authenticity of the image more accurately.

[0034] <2-1. System Configuration> Next, we will describe the configuration for realizing the above method. Figure 1 is a diagram showing an example of an image processing system configuration to which this technology is applied. The image processing system 100 shown in Figure 1 is a system in which an imaging device captures an image of a subject to generate an image, and the image is registered with a server. As shown in Figure 1, the image processing system 100 has an imaging device 111, a server 112, and a terminal device 113.

[0035] The imaging device 111, the server 112, and the terminal device 113 are connected to each other via a network 110 so that they can communicate with one another. The network 110 is a communication network that serves as the communication medium between each device. The network 110 may be a wired communication network, a wireless communication network, or both. For example, it may be a wired LAN (Local Area Network), a wireless LAN, a public telephone network, a wide-area communication network for wireless mobile devices such as so-called 4G or 5G lines, or the Internet, or a combination thereof. Furthermore, the network 110 may be a single communication network or multiple communication networks. Also, for example, part or all of the network 110 may be composed of communication cables of a predetermined standard, such as USB (Universal Serial Bus) (registered trademark) cables or HDMI (High-Definition Multimedia Interface) (registered trademark) cables.

[0036] In Figure 1, one imaging device 111, one server 112, and one terminal device 113 are shown, but the number of each of these devices is arbitrary.

[0037] The imaging device 111 captures an image of the subject and generates an image (captured image). The imaging device 111 also acquires 3D information on the same optical axis as the image. Furthermore, the imaging device 111 generates a signature (digital signature) of the information, including at least the image and the 3D information. For example, the imaging device 111 generates the signature using key information corresponding to the imaging device 111 (e.g., the device's private key). The imaging device 111 then transmits (uploads) the generated image, 3D information, and signature to the server 112.

[0038] Server 112 receives the information uploaded from the imaging device 111 and performs processing related to verifying the authenticity of the images. Terminal device 113 is a device operated by the user (verifier) ​​who verifies the authenticity of the images. Verification of image authenticity can be performed by server 112 or by the user of terminal device 113. When server 112 performs the process of verifying the authenticity of the images, server 112 displays the verification result on terminal device 113, thereby presenting the result to the user of terminal device 113. Also, when the user of terminal device 113 performs the task of verifying the authenticity of the images, server 112 displays information that assists in the verification on terminal device 113, thereby presenting that information to the user of terminal device 113.

[0039] <2-1-1. Imaging device> The imaging device 111 is comprised of an information processing terminal device having imaging and 3D information generation functions, such as a digital camera, smartphone, tablet terminal, or notebook personal computer. The imaging device 111 may be comprised of one device (electronic device) or multiple devices (electronic devices). For example, the imaging device 111 may be comprised of a digital camera and a smartphone. In that case, for example, the digital camera may acquire images and 3D information, and the smartphone may generate signatures for them and upload them to the server 112. In the following description, the imaging device 111 will be described as being comprised of a single device (electronic device).

[0040] Figure 2 is a block diagram showing an example of the configuration of an imaging device 111, which is one embodiment of an image processing device to which this technology is applied.

[0041] Note that Figure 2 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the imaging device 111 may have processing units that are not shown as blocks in Figure 2, or processes and data flows that are not shown as arrows or other symbols in Figure 2.

[0042] As shown in Figure 2, the imaging device 111 has a control unit 121 and an imaging processing unit 122. The control unit 121 controls each processing unit within the imaging processing unit 122. The imaging processing unit 122 is controlled by the control unit 121 and performs processing related to imaging.

[0043] The imaging processing unit 122 includes a storage unit 131, a key processing unit 132, an upload unit 133, and a recording unit 134.

[0044] The storage unit 131 has any storage medium, such as semiconductor memory or hard disk, and stores information in that storage medium. For example, the storage unit 131 pre-stores a device-specific ID corresponding to the imaging device 111. The device-specific ID is unique identification information for an electronic device used to identify that electronic device. In other words, the storage unit 131 pre-stores the ID assigned to the imaging device 111 (the ID for identifying the imaging device 111). Here, "pre-stored" refers to the initial state of the imaging device 111 or a state close to the initial state, such as when it is shipped from the factory.

[0045] The key processing unit 132 performs processing related to key information corresponding to electronic devices. For example, the key processing unit 132 reads a device-specific ID from the storage unit 131 and uses that device-specific ID to generate a device secret key, which is the private key corresponding to the imaging device 111, and a device public key, which is the public key corresponding to the imaging device 111. In other words, the key processing unit 132 can also be called a key generation unit. These device secret key and device public key are also called a pair key. The key processing unit 132 also supplies the pair key to the storage unit 131 and stores it in its storage medium. Furthermore, the key processing unit 132 supplies the device public key to either or both of the upload unit 133 and the recording unit 134 and provides it to the server 112. The key processing unit 132 may also generate a common key to be shared with the server 112 instead of this pair key. In that case, the key processing unit 132 supplies the generated common key to the storage unit 131 and stores it in its storage medium. Furthermore, the key processing unit 132 supplies the shared key to either or both of the upload unit 133 and the recording unit 134, causing them to provide it to the server 112. The key processing unit 132 may generate key information (paired key or shared key) without using a device-specific ID. For example, the key processing unit 132 may generate key information using random numbers.

[0046] The upload unit 133 has a communication function and can communicate with other devices via the network 110. For example, the upload unit 133 transmits (uploads) key information (device public key or shared key) supplied from the key processing unit 132 to the server 112. In other words, the upload unit 133 can also be called a providing unit (transmitting unit) that provides key information to the server 112.

[0047] The recording unit 134 has a drive that drives removable recording media such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and performs writing and reading operations. The recording unit 134 records information on the removable recording media via its drive. For example, the recording unit 134 records key information (device public key or common key) supplied from the key processing unit 132 on the removable recording media. For example, this removable recording media is mounted on the drive of another information processing device, the recorded key information is read, and transmitted to the server 112. In other words, in this case, the key information is provided to the server 112 via this removable recording media. Therefore, the recording unit 134 can also be said to be a provider unit that provides key information to the server 112. Note that the recording unit 134 may also record information on a non-removable recording media. For example, the recording unit 134 may record key information (device public key or common key) supplied from the key processing unit 132 on a non-removable recording media.

[0048] Furthermore, the imaging processing unit 122 includes an optical system 141, an image sensor 142, a RAW processing unit 143, a YUV processing unit 144, a reduced image generation unit 145, a metadata addition unit 146, a hash processing unit 147, a 3D information sensor 148, a 3D information processing unit 149, a signature generation unit 150, an image file generation unit 151, a signature control unit 153, and a confidence calculation unit 154. The image sensor 142 and the 3D information sensor 148 are collectively referred to as the sensor unit 161.

[0049] The optical system 141 is composed of optical elements such as lenses, mirrors, filters, and apertures, and exerts a predetermined influence on the light rays from the subject, guiding those light rays to the sensor unit 161. In other words, the light rays from the subject enter the sensor unit 161 via this optical system 141.

[0050] The image sensor 142 acquires an image of a subject by capturing an optical image of the subject. In other words, the image sensor 142 can also be called an image acquisition unit. For example, the image sensor 142 has a pixel array in which pixels having photoelectric conversion elements are arranged in a matrix. The image sensor 142 receives light rays from the subject incident via the optical system 141 in its pixel array, performs photoelectric conversion, and generates an image (RAW image). The image sensor 142 supplies the generated RAW image to the RAW processing unit 143.

[0051] The RAW processing unit 143 acquires the RAW image supplied from the image sensor 142 and performs predetermined processing on the RAW image. The content of this processing is arbitrary. For example, it may be correction of defective pixels for which normal pixel values ​​could not be obtained or noise reduction processing. The RAW processing unit 143 supplies the processed RAW image to the YUV processing unit 144.

[0052] The YUV processing unit 144 acquires the RAW image supplied by the RAW processing unit 143 and converts the RAW image into an image consisting of luminance components and chrominance components (also called a luminance chrominance image). For example, the YUV processing unit 144 performs color separation processing on the RAW image (for example, demosaicing processing in the case of a mosaic color filter such as a Bayer array) and converts the resulting color-separated RGB plane image into a luminance chrominance image. The YUV processing unit 144 also performs white balance correction on the color-separated RGB plane image or the converted luminance chrominance image.

[0053] The luminance chrominance image may be a YUV image consisting of a luminance component (Y) and chrominance components (U, V), or a YCbCr image consisting of a luminance component (Y) and chrominance components (Cb, Cr). In the following explanation, a YUV image will be used as an example of a luminance chrominance image. The YUV processing unit 144 supplies the obtained luminance chrominance image (YUV image) to the reduced image generation unit 145.

[0054] The reduced image generation unit 145 acquires the YUV image supplied from the YUV processing unit 144 and generates a reduced image of it. The method of generating this reduced image is arbitrary. For example, the reduced image may be generated by downsampling some of the pixel values ​​of the image, or by reducing the number of pixels by combining pixel values ​​for predetermined sub-regions. Alternatively, for example, it may be created by looking at all the pixels of the original image. If there are not enough taps, the reduced image generation unit 145 may repeat the reduction process multiple times at a magnification that provides enough taps to generate a reduced image at the desired magnification. The original YUV image is also referred to as the "original image" in relation to this reduced image. The reduced image generation unit 145 supplies the generated reduced image, along with the original image (YUV image), to the metadata addition unit 146.

[0055] The metadata addition unit 146 acquires the main image and the reduced image supplied from the reduced image generation unit 145. The metadata addition unit 146 generates metadata and associates it with the main image. The content of this metadata is arbitrary. For example, this metadata may include items defined by standards or items set by the manufacturer. The metadata addition unit 146 supplies the generated metadata, along with the main image and the reduced image, to the hash processing unit 147.

[0056] The hash processing unit 147 acquires the main image, reduced image, and metadata supplied from the metadata addition unit 146. The hash processing unit 147 also acquires 3D information supplied from the 3D information processing unit 149. The hash processing unit 147 calculates a hash value using the main image, reduced image, metadata, and 3D information. The hash processing unit 147 supplies the calculated hash value, along with the main image, reduced image, metadata, and 3D information, to the signature generation unit 150.

[0057] The hash processing unit 147 may be controlled and driven by the signature control unit 153. In other words, if the signature control unit 153 instructs the hash processing unit 147 to calculate a hash value, it calculates the hash value as described above, and if the signature control unit 153 does not instruct the hash processing unit 147 to calculate a hash value, it may omit the calculation of the hash value. If the calculation of the hash value is omitted, the hash processing unit 147 supplies the main image, reduced image, metadata, and 3D information to the signature generation unit 150.

[0058] The 3D information sensor 148 acquires 3D information from the optical image of the subject using the same optical axis as the image obtained by the image sensor 142. In other words, the 3D information sensor 148 can also be called a 3D information acquisition unit.

[0059] Here, the optical axis refers to the principal ray that passes through the center of the light beam passing through the entire system in an optical system. "Acquiring 3D information from the optical image of the subject on the same optical axis as the image" means that the optical axis of the optical image from which the image is obtained and the optical axis of the optical image from which the 3D information is obtained are the same. In other words, in this case, the optical image from the subject incident on the 3D information sensor 148 and the optical image from the subject incident on the image sensor 142 are incident on the imaging device 111 from the same point, pass through the optical system 141 along the same path, and are incident on the sensor unit 161. Therefore, 3D information is obtained from the same angle to the subject as in the image, for subjects within the range (angle of view) of the scene included in the image. For this reason, for example, if the image sensor and the distance measuring sensor were on different optical axes, it would be difficult to perform trick photography such as placing a mirror tilted 45 degrees only in front of the distance measuring sensor to take a photograph of a face, and then having the distance measuring sensor detect the distance to a person other than the subject in the photograph. In other words, the likelihood that the subject in the RAW image and the subject shown in the 3D information are the same increases.

[0060] Furthermore, the 3D information may include distance-related information for multiple locations within the image obtained by the image sensor 142 (i.e., multiple locations in the optical image from the subject), or information generated based on that distance-related information. Note that the "distance-related information" may be information indicating the distance from the imaging device 111 (3D information sensor 148) to the subject, or it may be information for deriving that distance. For example, the distance-related information may include a depth map, phase difference data, ToF data, or a collection of disparity images.

[0061] The 3D information sensor 148 supplies the generated 3D information to the 3D information processing unit 149.

[0062] In Figure 2, the image sensor 142 and the 3D information sensor 148 are configured as separate components, but the image sensor 142 and the 3D information sensor 148 may be integrated, or the image sensor 142 may also function as the 3D information sensor 148.

[0063] For example, if the distance-related information is ToF data, the 3D information sensor 148 may be configured as a ToF sensor that measures distance using the ToF method, separate from the image sensor 142. Also, if the distance-related information is phase difference data, the 3D information sensor 148 may be configured as a phase difference sensor that detects phase difference data, separate from the image sensor 142. In these examples, when the image sensor 142 and the 3D information sensor 148 are configured as separate units, one optical image (optical image from the subject) incident on the sensor unit 161 may be split into two optical images (identical optical images to each other) using a beam splitter (half mirror) with a prism or the like, with one optical image incident on the image sensor 142 and the other optical image incident on the 3D information sensor 148. In other words, in this case, an image is acquired from one of the two split optical images, and 3D information is acquired from the other optical image.

[0064] Furthermore, for example, if distance-related information is phase difference data, the 3D information sensor 148 may be composed of image plane phase difference detection pixels 171 formed in the effective pixel area of ​​the pixel array of the image sensor 142, as shown in Figure 3. The image plane phase difference detection pixels 171 are pixels that can also be used for phase difference autofocus functions and can detect phase difference data. In other words, in this case, the image sensor 142 and the 3D information sensor 148 are formed as a single unit. The 3D information sensor 148 (image plane phase difference detection pixels 171) acquires 3D information (phase difference data) using the phase difference method. Note that in Figure 3, only one image plane phase difference detection pixel is labeled, but all pixels indicated by black squares are image plane phase difference detection pixels 171. In other words, in this example, the 3D information sensor 148 (image plane phase difference detection pixels 171) acquires phase data for multiple locations in the image acquired by the image sensor 142. As in this example, when the image sensor 142 and the 3D information sensor 148 are configured as a single unit, one optical image (an optical image from the subject) incident on the sensor unit 161 is incident on the integrated image sensor 142 and 3D information sensor 148. In other words, in this case, both image and 3D information are acquired from that single optical image.

[0065] Furthermore, if the distance-related information is a collection of disparity images generated by 3D swing panoramic shooting or the like, the image sensor 142 also functions as the 3D information sensor 148. In other words, in this case, the image sensor 142 acquires the image and 3D information, and the 3D information sensor 148 can be omitted. That is, even in this example, the image and 3D information are acquired from a single optical image.

[0066] In the following explanation, unless otherwise specified, the 3D information sensor 148 will be described using the example where it is composed of image plane phase difference detection pixels 171. In other words, the explanation will be described using the example where phase difference data is detected as distance-related information by the 3D information sensor 148.

[0067] The 3D information processing unit 149 acquires 3D information supplied from the 3D information sensor 148. The 3D information processing unit 149 performs predetermined processing on the 3D information. For example, the 3D information processing unit 149 supplies the acquired 3D information to the hash processing unit 147. Also, based on the control of the signature control unit 153, if no signature is generated, the 3D information processing unit 149 reduces the resolution of the 3D information and supplies the reduced-resolution 3D information to the hash processing unit 147. Also, based on the control of the signature control unit 153, if a signature is generated, the 3D information processing unit 149 omits the reduction of the resolution of the 3D information and supplies the acquired 3D information to the hash processing unit 147. In other words, the 3D information processing unit 149 can also be called a 3D information resolution setting unit. Furthermore, the 3D information processing unit 149 determines whether the object (distance measurement target) indicated by the acquired 3D information is a plane or not, and supplies the determination result to the signature control unit 153. In other words, the 3D information processing unit 149 can also be called a plane determination unit.

[0068] The signature generation unit 150 generates a signature (electronic signature) corresponding to the image obtained by the image sensor 142 (main image) and the 3D information obtained by the 3D information sensor 148. This signature only needs to correspond to the main image and the 3D information, and may also correspond to information other than the main image and the 3D information. In other words, the signature generation unit 150 generates a signature for information that includes at least the main image and the 3D information. The signature generation unit 150 may generate this signature using key information corresponding to the imaging device 111. This key information may be, for example, a device secret key corresponding to the imaging device 111, or a common key shared with the server 112, etc. (a method in which the signing side and the verification side use the same key).

[0069] For example, the signature generation unit 150 obtains the main image, reduced image, metadata, 3D information, and hash value supplied from the hash processing unit 147. The signature generation unit 150 also obtains the device secret key corresponding to the imaging device 111, which is stored in the storage unit 131. Then, the signature generation unit 150 generates a signature by encrypting the hash value using the device secret key. In other words, in this example, the signature generation unit 150 generates signatures (digital signatures) for the main image, 3D information, reduced image, and metadata. The signature generation unit 150 supplies the generated signature along with the main image, reduced image, metadata, and 3D information to the image file generation unit 151.

[0070] Furthermore, this signature (at least the signature corresponding to the main image and 3D information) may be generated using an image other than the main image (YUV image). For example, instead of the main image, this signature may be generated using a reduced image (e.g., a screen nail of display size) obtained by reducing the YUV image generated by the YUV processing unit 144 to a extent that does not affect its features. In that case, the hash processing unit 147 reduces the main image (YUV image) to generate a screen nail, and obtains the screen nail, the reduced image, metadata, 3D information, and hash value. Then, the signature generation unit 150 generates the signature by encrypting the hash value using the device's secret key. The features of this screen nail (reduced image) are substantially equivalent to those of the YUV image generated by the YUV processing unit 144. Therefore, it can be said that the signature generated using this screen nail (reduced image) corresponds to the main image. By using a screen nail (reduced image), the increase in the amount of data for the hash value and signature can be suppressed.

[0071] Furthermore, the signature generation unit 150 may be controlled and driven by the signature control unit 153. In other words, if the signature generation unit 150 is instructed to generate a signature by the signature control unit 153, it will generate a signature as described above, and if the signature generation unit 153 is not instructed to generate a signature, it may omit the generation of the signature. If the generation of the signature is omitted, the signature generation unit 150 will supply the main image, a reduced image, metadata, and 3D information to the image file generation unit 151.

[0072] The image file generation unit 151 obtains the main image, thumbnail, metadata, 3D information, and signature supplied from the signature generation unit 150. The image file generation unit 151 compresses and encodes the YUV image, which is the main image, and converts it into a JPEG (Joint Photographic Experts Group) image. The compression encoding method is arbitrary. That is, the file format of the compressed and encoded main image is arbitrary and does not have to be JPEG. The image file generation unit 151 generates an image file in a predetermined format and stores the main image (JPEG image), thumbnail, metadata, 3D information, and signature in the image file. The compression encoding of the YUV image may be omitted, and the YUV image may be stored in the image file. The image file generation unit 151 supplies the image file to either or both of the upload unit 133 and the recording unit 134, and provides it to the server 112.

[0073] The upload unit 133 retrieves the image file supplied from the image file generation unit 151 and sends (uploads) that image file to the server 112. In other words, the upload unit 133 can also be described as a providing unit (transmitting unit) that provides image files to the server 112.

[0074] The recording unit 134 acquires the image file supplied from the image file generation unit 151 and records the image file on a removable recording medium. In other words, in this case, the image file is provided to the server 112 via this removable recording medium. Therefore, the recording unit 134 can also be said to be a providing unit that provides the image file to the server 112. Note that the recording unit 134 may also record the image file supplied from the image file generation unit 151 on a non-removable recording medium.

[0075] The signature control unit 153 controls whether or not to generate a signature. For example, the signature control unit 153 controls whether or not to generate a signature based on instructions based on user operations input via the control unit 121 or instructions from an application. The signature control unit 153 also controls whether or not to generate a signature based on the determination result of whether or not the object (distance measurement target) indicated by the 3D information processing unit 149 is a plane. For example, if the signature control unit 153 determines that the object (distance measurement target) indicated by the 3D information is a plane, it controls the signature to omit signature generation, and if it determines that the object (distance measurement target) indicated by the 3D information is not a plane, it controls the signature to generate a signature. The signature control unit 153 controls whether or not to generate a signature by controlling the hash processing unit 147 and the signature generation unit 150. For example, if the signature control unit 153 controls not to generate a signature, it causes the hash processing unit 147 to omit calculating the hash value and the signature generation unit 150 to omit signature generation. Furthermore, when the signature control unit 153 controls the system to generate a signature, it instructs the hash processing unit 147 to calculate a hash value and the signature generation unit 150 to generate a signature. When the signature control unit 153 controls the system not to generate a signature, it may instruct the hash processing unit 147 to calculate a hash value and the signature generation unit 150 to omit signature generation. The signature control unit 153 can also supply control information to the 3D information processing unit 149 indicating whether or not to generate a signature.

[0076] The reliability calculation unit 154 calculates the reliability of the 3D information generated by the 3D information sensor 148. In other words, the reliability calculation unit 154 can also be called a reliability generation unit that generates the reliability of the 3D information. For example, the reliability calculation unit 154 acquires metadata generated by the image sensor 142 and 3D information generated by the 3D information sensor 148. The reliability calculation unit 154 then compares the image and the 3D information and calculates the reliability of the 3D information (the certainty of the 3D information (shown by the bumps and depressions) relative to the image features (bumps and depressions)). Note that this method of calculating reliability is arbitrary. For example, the method described in International Publication No. 2019 / 073814 may be applied. The calculated reliability is supplied to the image file generation unit 151 via the 3D information sensor 148, the 3D information processing unit 149, the hash processing unit 147, and the signature generation unit 150. The image file generation unit 151 then stores the information indicating the reliability in the image file.

[0077] Alternatively, a RAW image may be stored as the main image in the image file instead of a compressed and encoded image (e.g., a JPEG image). In this case, the RAW image is supplied from the RAW processing unit 143 to the image file generation unit 151 via the YUV processing unit 144, the reduced image generation unit 145, the metadata addition unit 146, the hash processing unit 147, and the signature generation unit 150. The image file generation unit 151 stores the supplied RAW image as the main image in the image file. In this case, the YUV processing unit 144 may be omitted. Furthermore, both a RAW image and a compressed and encoded image (e.g., a JPEG image) may be stored as the main image in the image file. In this case, the signature may use a RAW image, a compressed and encoded image (e.g., a JPEG image), or both a RAW image and a compressed and encoded image (e.g., a JPEG image). Also, when a RAW image is used as the main image, the RAW image may be stored in a different file from other information (3D information, reduced images, metadata, signature, etc.). In that case, the file containing the RAW image and the file containing other information will each store information that links them together (for example, a UUID (Universal Unique Identifier)).

[0078] Furthermore, the storage of the reduced image into an image file may be omitted. In that case, the reduced image generation unit 145 may be omitted. Furthermore, the storage of the signature into an image file may be omitted. In that case, the signature generation unit 150 may be omitted.

[0079] <2-1-2. Server> Server 112 can have any configuration. For example, Server 112 may consist of a single information processing device or multiple information processing devices. Furthermore, Server 112 may be implemented as a cloud computing system (i.e., a cloud server) where processing is shared and collaboratively performed by multiple devices over a network.

[0080] Figure 4 is a block diagram showing an example of the configuration of a server 112, which is one embodiment of an image processing device to which this technology is applied.

[0081] Note that Figure 4 shows the main components such as processing units and data flows, and does not necessarily represent an exhaustive list. In other words, there may be processing units in server 112 that are not shown as blocks in Figure 4, or there may be processes or data flows that are not shown as arrows or other symbols in Figure 4.

[0082] As shown in Figure 4, the server 112 includes a control unit 201, a communication unit 221, an image analysis engine 222, and a device public key database 223. The control unit 201 controls the communication unit 221, the image analysis engine 222, and the device public key database 223.

[0083] The communication unit 221 has communication capabilities and can communicate with other devices via the network 110. For example, the communication unit 221 communicates with the imaging device 111 and receives key information (device public key or shared key) and image files transmitted from the imaging device 111. Also, when an information processing device reads key information and image files generated by the imaging device 111 from a removable recording medium and transmits them to the server 112, the communication unit 221 communicates with the information processing device and receives the key information and image files. The communication unit 221 then supplies the received information (e.g., key information or image files) to the image analysis engine 222, as in these examples. The communication unit 221 also transmits the results of the image authenticity verification and information assisting in the verification of image authenticity, etc., supplied by the image analysis engine 222, to the terminal device 113.

[0084] The image analysis engine 222 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and uses them to perform image analysis processing. The image analysis engine 222 also includes a device public key management unit 231, a signature verification unit 232, an image verification processing unit 233, a verification control unit 234, an optical axis determination unit 235, a reliability determination unit 236, a reliability calculation unit 237, a shutter speed determination unit 238, and a development processing unit 239.

[0085] The device public key management unit 231 manages the information stored in the device public key database 223. For example, the device public key management unit 231 registers the device public key supplied by the communication unit 221 in the device public key database 223. The device public key management unit 231 also reads a desired device public key from the device public key database 223. Furthermore, the device public key management unit 231 determines whether the device public key is valid. Therefore, the device public key management unit 231 can also be called a public key determination unit.

[0086] The device public key management unit 231 may also handle a common key (key information shared by the imaging device 111 and the server 112) instead of the device public key. In that case, the device public key management unit 231 registers the common key supplied by the communication unit 221 in the device public key database 223. The device public key management unit 231 also reads the desired common key from the device public key database 223. Of course, the device public key management unit 231 may also handle both the device public key and the common key. In other words, the device public key management unit 231 may handle key information (either the device public key or the common key, or both).

[0087] The signature verification unit 232 performs processing related to the verification of the signature stored in the image file supplied by the communication unit 221. This signature is the signature of the information stored in the image file, which includes at least the main image and 3D information. For example, the signature verification unit 232 uses key information (device public key or common key) corresponding to another device (e.g., imaging device 111) that is the source of the image and 3D information to verify the validity of the signature of the image and 3D information.

[0088] The image verification processing unit 233 performs processing related to verifying the authenticity of an image. For example, the image verification processing unit 233 verifies the authenticity of an image by comparing it with 3D information detected on the same optical axis as the image. Furthermore, if the signature verification unit 232 fails to verify the validity of the signature, the image verification processing unit 233 determines that the image is not authentic.

[0089] The image verification processing unit 233 includes an image verification unit 241 and a support processing unit 242. The image verification unit 241 performs processing to confirm the authenticity of the image, such as image verification. The support processing unit 242 performs support processing for confirming the authenticity of the image, such as presenting information to the verifier.

[0090] The verification control unit 234 performs control related to the verification of the authenticity of the image. For example, the verification control unit 234 controls the image verification processing unit 233 to determine whether or not to verify the authenticity of the image. For example, the verification control unit 234 controls whether or not to verify the authenticity of the image based on the determination result by the optical axis determination unit 235, the determination result by the reliability determination unit 236, or the determination result by the shutter speed determination unit 238, etc.

[0091] The optical axis determination unit 235 determines whether the 3D information was acquired on the same optical axis as the image. For example, the optical axis determination unit 235 makes this determination based on information stored as metadata in the image file that indicates whether the 3D information was acquired on the same optical axis as the image. This "information indicating that the 3D information was acquired on the same optical axis as the image" may be, for example, flag information indicating whether the image and 3D information were obtained on the same optical axis, or it may be the name of the equipment, model name, or identification information of the equipment that always obtains the image and 3D information on the same optical axis.

[0092] The reliability determination unit 236 determines whether the 3D information is reliable based on reliability information indicating the reliability of the 3D information, which is stored as metadata related to the image in the image file that stores the image and the 3D information.

[0093] The reliability calculation unit 237 calculates the reliability of the 3D information based on the camera parameters related to the image, which are stored in the image file containing the image and 3D information. These camera parameters can be any information. For example, the number of effective pixels, F-number, or focal length of the sensor unit 161 (image sensor 142 or 3D information sensor 148) of the imaging device 111 may be included as camera parameters. Furthermore, the method for calculating the reliability is arbitrary.

[0094] The shutter speed determination unit 238 determines whether the shutter speed used when the image was generated is faster than a predetermined standard, based on camera parameters related to the image stored in the image file that stores the image and 3D information.

[0095] The development processing unit 239 performs development processing to convert the RAW image contained in the image file into a YUV image.

[0096] The device public key database 223 has a storage medium such as a hard disk or semiconductor memory, and stores information such as the device public key on that storage medium. Figure 5 is a diagram showing an example of the information stored in the device public key database 223. For example, the device public key database 223 stores information such as the device-specific ID, device public key, and invalidation date in relation to each other. The invalidation date indicates the date on which the device public key is invalidated. Note that the device public key database 223 may store a common key (key information shared by the imaging device 111 and the server 112) instead of the device public key. Of course, the device public key database 223 may store both the device public key and the common key. In other words, the device public key database 223 may store key information (either one or both of the device public key and the common key).

[0097] In the following, unless otherwise specified, the images and 3D information processed by the image analysis engine 222 are assumed to have been detected on the same optical axis. In other words, only the image files generated by the imaging device 111 are provided to the server 112.

[0098] <2-1-3. Terminal Devices> Figure 6 is a block diagram showing an example of the configuration of a terminal device 113, which is one embodiment of an image processing apparatus to which this technology is applied.

[0099] Note that Figure 6 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, there may be processing units in the terminal device 113 that are not shown as blocks in Figure 6, or there may be processes or data flows that are not shown as arrows or other symbols in Figure 6.

[0100] As shown in Figure 6, the terminal device 113 includes a CPU 301, ROM 302, RAM 303, bus 304, input / output interface 310, input unit 311, output unit 312, storage unit 313, communication unit 314, and drive 315.

[0101] The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output interface 310 is also connected to the bus 304. An input / output interface 310 is connected to an input unit 311, an output unit 312, a storage unit 313, a communication unit 314, and a drive 315.

[0102] The input unit 311 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 312 consists of, for example, a display, speaker, and output terminals. The storage unit 313 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 314 consists of, for example, a network interface. The drive 315 drives a removable recording medium 321 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0103] For example, the CPU 301 loads programs stored in the ROM 302 and memory unit 313 into the RAM 303 and executes them. The RAM 303 also stores data necessary for the CPU 301 to perform various processes as appropriate. By executing programs in this way, the CPU 301 performs processes such as presenting information related to image verification.

[0104] The program executed by the computer may be recorded on a removable recording medium 321, such as a packaged medium, and provided to the terminal device 113. In that case, the program is read from the removable recording medium 321 mounted on the drive 315 and installed in the storage unit 313 via the input / output interface 310.

[0105] Alternatively, this program may be provided to the terminal device 113 via a wired or wireless transmission medium, such as a local area network, the internet, or digital satellite broadcasting. In that case, the program is received by the communication unit 314 and installed in the storage unit 313.

[0106] In addition, this program may be pre-installed in ROM302 or memory unit313.

[0107] <2-2. Processing of the imaging device> <2-2-1. Key Processing> Next, we will explain each process performed by the imaging device 111, etc. First, we will explain an example of the key processing flow in which the imaging device 111 generates a device public key and uploads it to the server 112, referring to the flowchart in Figure 7.

[0108] When key processing is initiated, the key processing unit 132 of the imaging device 111 reads the device-specific ID stored in the storage unit 131 in step S101 and generates a pair of keys (device secret key and device public key) using that device-specific ID. In other words, the key processing unit 132 generates a device secret key and a device public key corresponding to the imaging device 111.

[0109] As shown in Figure 8, the imaging device 111 has a display device 401 such as an LCD (Liquid Crystal Display) or an OELD (Organic Electro Luminescence Display). The imaging device 111 also has an operating device 402 such as buttons or a directional pad.

[0110] For example, a user of the imaging device 111 operates the operating device 402 to display a GUI (Graphical User Interface) as shown in Figure 8 on the display device 401. The GUI in the example in Figure 8 is a menu screen for selecting the process to execute. When the user operates the operating device 402 to move the cursor and selects the "Create Pair Key" menu, the process in step S101 is executed, and a pair key is generated as described above. Once the pair key is generated, a GUI as shown in Figure 9 is displayed on the display device 401.

[0111] In step S102, the key processing unit 132 supplies the pair of keys (device secret key and device public key) generated in step S101 to the storage unit 131 for storage. The storage unit 131 stores the pair of keys supplied by the key processing unit 132.

[0112] Figure 10 shows an example of information stored in the storage unit 131. The storage unit 131 has a device-specific ID 411 pre-stored in it, and when the process in step S102 is executed, the device secret key 412 and the device public key 413 are further stored in it.

[0113] In step S103, the key processing unit 132 supplies the device public key generated in step S101 to the upload unit 133, which then transmits (uploads) it to the server 112. The upload unit 133 uploads the device public key supplied by the key processing unit 132 to the server 112 via the network 110. In other words, the device public key is provided to the server 112.

[0114] For example, in the example in Figure 9, when the user operates the control device 402 and presses the OK button on the GUI displayed on the display device 401, the GUI shown in Figure 11 is displayed on the display device 401. In other words, the "Generate Pair Key" option becomes unavailable on the menu screen in the example in Figure 8.

[0115] Then, when the user operates the control device 402 to move the cursor and selects the "Output Public Key" menu as shown in Figure 12, the process in step S103 is executed, and the device public key is uploaded as described above. Once the device public key is uploaded, a GUI as shown in Figure 13 is displayed on the display device 401.

[0116] In step S111, the communication unit 221 of server 112 receives the uploaded device public key.

[0117] In step S112, the device public key management unit 231 registers the device public key received in step S111 in the device public key database 223. At that time, the device public key management unit 231 registers the device public key in the device public key database 223 in association with the device-specific ID of the imaging device 111, which is the source of the device public key. The server 112 knows the device-specific ID of the imaging device 111, which is the source of the device public key. For example, when the server 112 starts a session with the imaging device 111, it obtains the device-specific ID from the imaging device 111. Alternatively, the device-specific ID may be uploaded in association with the device public key. Furthermore, the device public key management unit 231 may set an invalidation date on which the device public key becomes invalid and register that invalidation date in association with the device public key in the device public key database 223.

[0118] Once step S112 is complete, the key processing is finished. By executing each process in this manner, the imaging device 111 generates a pair of keys corresponding to itself and uploads the device's public key to the server 112. This allows the server 112 to detect tampering with images and 3D information using the signature. Therefore, the server 112 can more accurately determine the authenticity of the images.

[0119] In the above description, the device public key is uploaded from the imaging device 111 to the server 112 via communication. However, the device public key generated in the imaging device 111 may be recorded on a removable recording medium and provided to the server 112 via that removable recording medium. In that case, for example, in step S103, the key processing unit 132 supplies the generated device public key to the recording unit 134. The recording unit 134 records the device public key on a removable recording medium or the like. For example, this removable recording medium is mounted on the drive of another information processing device, the recorded key information is read, and it is transmitted to the server 112. In step S111, the communication unit 221 of the server 112 receives the transmitted device public key.

[0120] Furthermore, the timing of providing the device public key to server 112 is arbitrary. For example, the device public key may be provided to server 112 at the same time as the image (for example, stored in the image file). Alternatively, the device public key may be provided to server 112 before the image file. Furthermore, the device public key may be provided to server 112 after the image file.

[0121] Furthermore, the server 112 (device public key management unit 231) may verify the validity of the provided device public key. For example, the imaging device 111 and the server 112 may have a common key, and the imaging device 111 encrypts the device public key using that common key and provides the encrypted device public key to the server 112. The server 112 then verifies the validity of the device public key by decrypting the encrypted device public key using the common key. The validity of the device public key may be verified in this way. Alternatively, the imaging device 111 and the server 112 may have a common key, and the imaging device 111 calculates the hash value of the device public key using that common key and provides the hash value to the server 112. The server 112 then verifies the validity of the device public key by verifying the hash value using the common key. The validity of the device public key may be verified in this way. Alternatively, the imaging device 111 may generate a signature for the device public key using a model-specific secret key, which is a secret key corresponding to the model, and provide the signature to the server 112. The server 112 then verifies the validity of the device public key by verifying its signature using the device public key (i.e., the public key corresponding to the device), which is the public key corresponding to the device's private key. The validity of the device public key may be verified in this way. Alternatively, the imaging device 111 encrypts the device public key using the server public key, which is the public key corresponding to the server 112, and provides the encrypted device public key to the server 112. The server 112 then verifies the validity of the device public key by decrypting the encrypted device public key using the server private key, which is the private key corresponding to the server 112. The validity of the device public key may be verified in this way. Alternatively, a trusted person or organization may register the device public key in the device public key database 223 of the server 112.

[0122] In the above explanation, we described the case where the imaging device 111 provides the server 112 with a public device key, but a common key may be provided instead of the public device key. Alternatively, both the public device key and the common key may be provided. In other words, key information (either the public device key or the common key, or both) may be provided from the imaging device 111 to the server 112.

[0123] <2-2-2. Preview Confidence Determination Process> The reliability of the acquired 3D information obtained by the 3D information sensor 148 is calculated and may be displayed on the display device 401 as a preview screen. Acquired 3D information is 3D information corresponding to the acquired image obtained by the 3D information sensor 148 during the period when the image sensor 142 is driven in acquisition mode. Acquisition mode is the operating mode in which the image sensor 142 acquires the acquired image. This acquired 3D information is acquired on the same optical axis as the acquired image.

[0124] An example of the flow of the preview reliability determination process, which displays a preview screen including the reliability of such acquired 3D information, will be explained with reference to the flowchart in Figure 14.

[0125] When the preview reliability determination process is started, the sensor unit 161 of the imaging device 111 is driven in acquisition mode in step S131, and the acquired image and acquired 3D information are acquired on the same optical axis. In other words, the image sensor 142 acquires the acquired image from the optical image from the subject. The 3D information sensor 148 acquires the acquired 3D information from the optical image from the subject on the same optical axis as the acquired image.

[0126] For example, in acquisition mode, the image sensor 142 generates the acquired image 421 shown in Figure 15, and the 3D information sensor 148 generates the acquired 3D information 422 shown in Figure 15. In the acquired 3D information 422, each polygon represents distance-related information (e.g., phase data or depth value). The fewer the number of sides of each polygon, the further away the subject in that region is from the imaging device 111 (3D information sensor 148). In other words, the more sides of each polygon, the closer the subject in that region is to the imaging device 111 (3D information sensor 148). The resolution of the acquired 3D information 422 is arbitrary, but if the amount of data is not considered, a higher resolution (e.g., the same resolution as the acquired image 421) is desirable.

[0127] Since the captured image 421 and the captured 3D information 422 correspond to each other, for example, when the captured image 421 and the captured 3D information 422 are superimposed, as in the superimposed image 423, the topography shown by the captured 3D information 422 basically matches the topography shown by the captured image 421 (the topography estimated from the captured image 421). However, the 3D information sensor 148 cannot always detect correct distance-related information. For example, incorrect distance-related information may be obtained depending on the composition and angle of the subject. In such cases, the topography may not match (or the degree of match may be reduced) between the captured 3D information 422 and the captured image 421. In such a state, when the subject is imaged and an image (captured image) and 3D information are generated, the topography may not match (or the degree of match may be reduced) between the image and the 3D information, similar to the case of the captured image 421 and the captured 3D information 422. Furthermore, if the authenticity of an image is verified using 3D information whose surface features do not match (or do not match to a high degree), the accuracy of the verification result may be reduced.

[0128] Therefore, the imaging device 111 calculates the reliability of the acquired 3D information and displays it on the display device 401 as a preview screen.

[0129] In other words, in step S132, the confidence calculation unit 154 analyzes the captured image acquired in step S131 and estimates the topography of the subject (also referred to as the topography of the captured image). The method for estimating the topography of the captured image is arbitrary. For example, the confidence calculation unit 154 may determine the features of the captured image and estimate the topography of the captured image based on those features. For example, the confidence calculation unit 154 may detect faces, eyes, noses, ears, etc., included in the captured image and estimate the topography of each detected part.

[0130] Then, in step S133, the confidence calculation unit 154 compares the surface topography of the acquired image (analysis result) estimated in step S132 with the surface topography indicated by the acquired 3D information (also referred to as the surface topography of the acquired 3D information), and calculates the confidence level of the 3D information based on the degree of agreement. The method for calculating this confidence level is arbitrary. For example, the method described in International Publication No. 2019 / 073814 may be applied.

[0131] In step S134, the control unit 121 displays the comparison result obtained from the processing in step S133 (the comparison result between the surface topography of the captured image estimated in step S132 and the surface topography of the 3D information) on the display device 401, including it in the preview screen. This comparison result can be any kind of information. For example, the control unit 121 may display the calculated confidence score as a numerical value or an image. Alternatively, the control unit 121 may compare the calculated confidence score with a predetermined threshold and display the comparison result (for example, whether the confidence score is higher or lower (compared to the threshold)) as text or an image. The captured image 421 and the captured 3D information 422 may also be displayed on the display device 401, including them in the preview screen.

[0132] Once step S134 is completed, the preview confidence level determination process ends.

[0133] By performing each process in this manner, the imaging device 111 can present the user with the reliability of the acquired 3D information on the preview screen. Based on this information, the user can perform imaging in a state that increases reliability, for example, by correcting the position and orientation of the imaging device 111. Therefore, the server 112 can determine the authenticity of the image more accurately.

[0134] <2-2-3. Image Processing> Next, an example of the imaging process flow performed by the imaging device 111 when imaging a subject will be explained with reference to the flowchart in Figure 16.

[0135] When the imaging process begins, the sensor unit 161 of the imaging device 111 acquires the RAW image and 3D information on the same optical axis in step S151. That is, the image sensor 142 acquires the RAW image of the subject by capturing an optical image of the subject. The 3D information sensor 148 acquires 3D information from the optical image of the subject on the same optical axis as the RAW image. The 3D information sensor 148 acquires 3D information at the start timing of the main exposure when the main exposure is started by the image sensor 142. Main exposure refers to the exposure for so-called "imaging" to obtain the image to be saved. For example, main exposure is started based on an imaging instruction operation by the user (e.g., fully pressing the shutter button). That is, exposure to obtain the captured image (exposure during the period when the device is driven in capture mode) is not included in main exposure. For example, even when the imaging device 111 captures an image of a subject in single autofocus mode (AF-S), the 3D information sensor 148 acquires and stores 3D information (phase difference information) at the start of the main exposure of the image sensor 142, as shown in the timing chart in Figure 17. Single autofocus mode (AF-S) refers to a mode in which the focal length is adjusted based on a predetermined operation by the user (e.g., half-pressing the shutter button) and then fixed. In this single autofocus mode, when a predetermined operation such as half-pressing the shutter button is performed by the user, the optical system 141 adjusts its focal length to focus on the subject based on the control of the control unit 121 and focuses on the subject. Once the subject is in focus, the control unit 121 performs control to fix the focus on the optical system 141. This control is also called focus lock. In other words, as shown in the timing chart in Figure 17, when the shutter button is half-pressed, focus lock is applied. When this focus lock is applied, the optical system 141 fixes its focal length.

[0136] In other words, if the focal length can be fixed for an extended period, such as in single autofocus mode, it becomes possible to perform trick photography by, for example, acquiring 3D information at the moment focus lock is applied, and then subsequently (while keeping the focal length fixed) changing the subject and taking another image. That is, such trick photography can generate false images in which the 3D information and the actual image have different subjects.

[0137] In contrast, by ensuring that 3D information is acquired at the start of the exposure, as described above, it becomes difficult to perform such trick photography. In other words, it becomes difficult to generate false images in which the subject differs between the 3D information and the image. Therefore, server 112 can more accurately verify the authenticity of the image.

[0138] For example, the image sensor 142 generates the image 431 in Figure 18, and the 3D information sensor 148 generates the 3D information 432 in Figure 18. In the 3D information 432, each polygon represents distance-related information (e.g., phase data, depth value, etc.). The fewer the number of sides of each polygon, the further away the subject in that region is from the imaging device 111 (3D information sensor 148). In other words, the more sides of each polygon, the closer the subject in that region is to the imaging device 111 (3D information sensor 148). The resolution of the 3D information 432 is arbitrary, but if the amount of data is not considered, a higher resolution (e.g., the same resolution as image 431) is desirable.

[0139] The superimposed image 433 in Figure 18 shows an example of the superimposed state of image 431 and 3D information 432. Since image 431 and 3D information 432 correspond, the surface irregularities shown by 3D information 432 basically match the surface irregularities of image 431 (the surface irregularities estimated from image 431), as shown in this superimposed image 433.

[0140] In step S152, the RAW processing unit 143 performs predetermined processing on the RAW image obtained in step S151. For example, the RAW processing unit 143 performs processing on the RAW image such as correcting defective pixels for which normal pixel values ​​could not be obtained and noise reduction processing.

[0141] In step S153, the YUV processing unit 144 converts the RAW image that has undergone predetermined processing in step S152 into a YUV image. For example, the YUV processing unit 144 performs color separation processing (for example, demosaicing processing in the case of a mosaic color filter such as a Bayer array) on the RAW image and converts the resulting color-separated RGB plane image into a YUV image. The YUV processing unit 144 also performs white balance correction on the color-separated RGB plane image or the converted YUV image.

[0142] In step S154, the reduced image generation unit 145 generates a reduced image by reducing the YUV image (the main image). For example, the reduced image generation unit 145 reduces image 431 to generate the reduced image 434 in Figure 18. The method of generating this reduced image is arbitrary. The size of the reduced image is also arbitrary. For example, the reduced image may be a so-called thumbnail or a screennail.

[0143] In step S155, the metadata addition unit 146 generates metadata and adds it to the image.

[0144] In step S156, the hash processing unit 147 calculates a hash value using the main image, 3D information, reduced image, and metadata.

[0145] In step S157, the signature generation unit 150 generates a signature for information including at least the main image and 3D information. For example, the signature generation unit 150 generates signatures for the main image, 3D information, reduced image, and metadata by encrypting the hash value calculated in step S156 using the device secret key corresponding to the imaging device 111. Alternatively, the signature generation unit 150 may generate this signature by encrypting the hash value using a common key instead of the device secret key.

[0146] In step S158, the image file generation unit 151 compresses and encodes the YUV image (main image) to generate a JPEG image. The image file generation unit 151 also generates an image file and stores the main image (JPEG image), a thumbnail, metadata, and a signature.

[0147] Figure 19 shows an example of the main structure of the image file. The image file 440 shown in Figure 19 contains the main image 431 and a reduced image 434. The main image 431 may be a JPEG image (compressed and encoded main image), a RAW image or a YUV image (uncompressed and encoded main image), or both. The reduced image 434 may be a JPEG image obtained by compressing and encoding a YUV image that has been reduced by resizing, or a reduced YUV image, or both.

[0148] When a RAW image is stored in the image file 440 as the main image 431, the RAW image acquired by the image sensor 142 is supplied to the image file generation unit 151 via the RAW processing unit 143, YUV processing unit 144, reduced image generation unit 145, metadata addition unit 146, hash processing unit 147, and signature generation unit 150. The image file generation unit 151 stores the RAW image in the image file 440 as the main image 431. When a JPEG image (compressed image) is stored in the image file 440 as the main image 431 along with a RAW image or YUV image (uncompressed image), the image file generation unit 151 compresses and encodes the YUV image to generate a JPEG image, and stores that JPEG image together with the RAW image or YUV image in the image file 440 as the main image 431.

[0149] Furthermore, when a JPEG image (compressed and encoded image) is to be stored in the image file 440 as a reduced image 434, the reduced image generation unit 145 reduces the YUV image. Then, the image file generation unit 151 compresses and encodes the reduced YUV image to generate a JPEG image, and stores that JPEG image in the image file 440 as a reduced image 434. Note that when a JPEG image (compressed and encoded image) is to be stored in the image file 440 as a reduced image 4341 along with a YUV image (uncompressed and encoded image), the image file generation unit 151 stores the JPEG image generated as described above, along with the reduced YUV image, in the image file 440 as a reduced image 434.

[0150] Furthermore, the image file 440 stores standard metadata 441 and additional metadata 442 as metadata. Standard metadata 441 consists of items defined by standards, for example. Additional metadata 442 consists of items not included in standard metadata 441, such as items set by the manufacturer. For example, additional metadata 442 may include a device-specific ID 411. Additional metadata 442 may also include shooting information 451, which is information related to the shooting of the subject.

[0151] Furthermore, the 3D information 432 is stored in the image file 440 as additional metadata. In addition, the signature generated in step S157 is stored in the image file 440 as additional metadata (signature 452).

[0152] Furthermore, information indicating that the image 431 and 3D information 432 were obtained on the same optical axis may be stored in the image file 440 (for example, as additional metadata). This information may be, for example, flag information indicating whether the image 431 and 3D information 432 were obtained on the same optical axis, or it may be the name of the equipment, model name, or identification information of the equipment on which the image 431 and 3D information 432 are always obtained on the same optical axis. By including such information in the image file 440, signature determination becomes possible without registering the equipment's public key. In other words, the image file generation unit 151 may store metadata in the image file that includes information indicating that the 3D information was acquired on the same optical axis as the image.

[0153] Furthermore, information indicating the shutter speed of the image sensor 142 during imaging of the subject for image generation may be stored in the image file 440 (for example, as additional metadata). This information may indicate the shutter speed in any way. For example, this information may indicate the shutter speed numerically. Alternatively, this information may indicate the shutter speed in steps within a predetermined level range. Furthermore, this information may indicate whether the shutter speed is faster or slower than a predetermined standard. In other words, the image file generation unit 151 may store the imaging shutter speed as metadata in the image file.

[0154] As described above, the generation of 3D information (detection of distance-related information) is performed at the timing when exposure begins during imaging by the image sensor 142. Therefore, if the shutter speed is slow, i.e., if the exposure period is long, it may be possible to perform trick photography such as changing the subject during exposure. In other words, it may be possible to generate a false image in which the subject differs between the 3D information and the image.

[0155] For example, if you set the shutter speed to 10 seconds in a dark room, point the imaging device 111 at a person until just before taking the picture, and then turn the imaging device 111 towards the composite image monitor after the exposure starts, you can obtain an image of a different subject from the subject shown in the 3D information.

[0156] Therefore, as described above, the image file generation unit 151 stores information indicating the shutter speed of the image capture when the image is generated as metadata in the image file. In this way, the server 112, which verifies the authenticity of the image, can determine the shutter speed at the time of image generation based on the metadata contained in the image file. In other words, the server 112 can verify the authenticity of the image by taking that shutter speed into consideration. Consequently, the server 112 can verify the authenticity of the image more accurately.

[0157] Furthermore, if both a RAW image and a JPEG image are stored in the image file 440 as the main image 431, the signature 452 may be the signature of the RAW image stored in the image file 440, or the signature of the JPEG image stored in the image file 440, or the signatures of both the RAW image and the JPEG image stored in the image file 440.

[0158] Returning to Figure 16, in step S159, the recording unit 134 records the image file generated in step S158 onto a removable recording medium or the like. This image file is then provided to the server 112 via this removable recording medium or the like.

[0159] In step S160, the upload unit 133 uploads the image file generated in step S158 to the server 112. In other words, the image file is provided to the server 112 via communication. For example, when a guidance screen as shown in Figure 20 is displayed on the display device 401, the user operates the operation device 402 and instructs the upload of an image file from a menu screen or the like. When this instruction is supplied to the upload unit 133 via the control unit 121, the upload unit 133 executes the process in step S160 and uploads the image file to the server 112.

[0160] The imaging process ends when step S160 is completed. Note that either step S159 or step S160 may be omitted.

[0161] By performing each process as described above, the imaging device 111 can provide the server 112 with the image generated by imaging the subject, the 3D information obtained on the same optical axis as the image, and the signature of the image and 3D information generated using the device secret key corresponding to the imaging device 111. This allows the server 112 to determine the authenticity of the image using the untampered image and 3D information. Therefore, the server 112 can determine the authenticity of the image more accurately.

[0162] Furthermore, if the RAW image and 3D information obtained by the processing in step S151, as well as the metadata generated by the processing in step S155, are saved, the processing in the other steps (processing of the RAW image, generation of a YUV image, generation of a reduced image, calculation of a hash value, generation of a signature, generation and provision of an image file, etc.) may be executed as processing separate from the imaging process after the imaging process is completed.

[0163] <2-2-4. Image Processing (Signature Control)> The imaging device 111 may be configured to control whether or not to create a signature. An example of the imaging process in this case will be explained with reference to the flowchart in Figure 21.

[0164] When the imaging process starts, the control unit 121 sets the signature execution mode (whether or not to generate a signature) in step S181 based on instructions from the user or from an application.

[0165] Each of the processes in steps S182 to S186 is performed in the same manner as the processes in steps S161 to S165 in Figure 16. That is, the main image, 3D information, reduced image, metadata, etc. are generated.

[0166] In step S187, the signature control unit 153 determines whether or not to generate a signature based on the signature execution mode set in step S181. If it is determined that the signature execution mode is a mode for generating a signature, the process proceeds to step S188.

[0167] Steps S188 and S189 are performed in the same manner as steps S166 and S167 in Figure 16. That is, a hash value is calculated for information including at least the main image and 3D information, and a signature is generated using the device secret key corresponding to the imaging device 111. In other words, a signature is generated for information including at least the main image and 3D information.

[0168] Once the processing in step S189 is completed, the process proceeds to step S190. Also, if it is determined in step S187 that the signature execution mode is a mode that does not generate a signature, the processing in steps S188 and S189 is omitted, and the process proceeds to step S190.

[0169] Each of the processes in steps S190 to S192 is executed in the same manner as the processes in steps S158 to S160 in Figure 16. That is, an image file is generated, and that image file is recorded on a removable recording medium or the like by the recording unit 134, or uploaded to the server 112.

[0170] When the processing in step S192 is completed, the imaging process is finished. In this case as well, either the processing in step S191 or step S192 may be omitted. Furthermore, if the RAW image and 3D information obtained by the processing in step S182, as well as the metadata generated by the processing in step S186, are saved, the processing in the other steps (setting the signature execution mode, processing the RAW image, generating a YUV image, generating a reduced image, calculating a hash value, generating a signature, generating or providing an image file, etc.) may be executed as separate processes from the imaging process after the imaging process is completed.

[0171] By performing each process as described above, the imaging device 111 can control whether or not to generate a signature for information including at least the image and 3D information. This allows the imaging device 111 to suppress the increase in load caused by generating a signature when a signature is not required.

[0172] If a signature is not generated, the resolution of the 3D information may be reduced. For example, if the authenticity of an image is verified using 3D information, the higher the resolution of the 3D information, the more accurately the server 112 can determine the authenticity of the image. However, the higher the resolution of the 3D information, the larger the amount of data it contains. Therefore, if 3D information is not used to verify the authenticity of an image and a signature for the 3D information is not required, the higher the resolution of the 3D information, the larger the amount of data in the image file will be. Thus, as described above, if a signature is not generated, the 3D information processing unit 149 (3D information resolution setting unit) may control the system to reduce the resolution of the 3D information. An example of the imaging process in this case will be explained with reference to the flowchart in Figure 22.

[0173] Once the imaging process begins, steps S211 to S216 are executed in the same manner as steps S181 to S186 in Figure 21. That is, the signature execution mode is set based on user instructions or instructions from an application, and the main image, 3D information, thumbnail image, metadata, etc., are generated.

[0174] In step S217, the signature control unit 153 determines whether or not to generate a signature based on the signature execution mode set in step S211. If it is determined that the signature execution mode is a mode for generating a signature, the process proceeds to step S218.

[0175] Steps S218 and S219 are performed in the same manner as steps S188 and S189 in Figure 21. That is, a hash value is calculated for information including at least the main image and 3D information, and a signature is generated using the device secret key corresponding to the imaging device 111. In other words, a signature is generated for information including at least the main image and 3D information. Once the process in step S219 is completed, the process proceeds to step S221.

[0176] Furthermore, if it is determined in step S217 that the signature execution mode is a mode that does not generate a signature, the process proceeds to step S220. In step S220, the 3D information processing unit 149 reduces the resolution of the 3D information. In other words, the 3D information processing unit 149 (3D information resolution setting unit) reduces the resolution of the 3D information if no signature is generated. Note that in this case, the resolution may include not only the resolution in the planar direction (resolution of the X and Y axes) but also the resolution of the dynamic range of each pixel value (i.e., the resolution in the depth direction). In other words, the resolution of the dynamic range of each pixel value may also be reduced. When the processing in step S220 is completed, the process proceeds to step S221.

[0177] Each of the processes in steps S221 to S223 is executed in the same manner as the processes in steps S190 to S192 in Figure 21. That is, an image file is generated, and that image file is recorded on a removable recording medium or the like by the recording unit 134, or uploaded to the server 112. However, the process in step S221 is executed as follows: If a signature is generated, the image file generation unit 151 stores the main image, 3D information, a reduced image, metadata, and the signature in the image file. If a signature is not generated, the image file generation unit 151 stores the main image, reduced-resolution 3D information, a reduced image, and metadata in the image file. When the process in step S223 is completed, the imaging process is completed. In this case as well, either the process in step S222 or the process in step S223 may be omitted. Furthermore, if the RAW image and 3D information obtained by the processing in step S212, as well as the metadata generated by the processing in step S216, are saved, the processing in the other steps (setting the signature execution mode, processing the RAW image, generating a YUV image, generating a reduced image, calculating a hash value, generating a signature, reducing the resolution of 3D information, generating or providing image files, etc.) may be executed as separate processes from the imaging process after the imaging process is completed.

[0178] By performing each process as described above, the imaging device 111 can reduce the resolution of the 3D information, at least when it does not generate a signature for the image and the information containing the 3D information. Therefore, it is possible to suppress an unnecessary increase in the amount of data in the image file.

[0179] If a signature is not generated, the 3D information may not be stored in the image file. In that case, the process in step S220 is omitted. In other words, if it is determined in step S217 that the signature execution mode is a mode that does not generate a signature, the process proceeds to step S221. In this case, in step S221, the image file generation unit 151 stores the main image, a thumbnail image, and metadata in the image file.

[0180] Furthermore, if the shape of the subject is flat, a signature may not be generated. When the shape of the subject is flat, it may be difficult to determine the authenticity of the image based on the surface irregularities of the subject in 3D information. In such cases, as described above, a signature may not be generated. By not generating a signature, the authenticity of the image can be determined to be lacking. In other words, the imaging device 111 may determine whether the shape of the subject is flat or not, and if the shape of the subject is flat, it may generate an image file such that the authenticity of the image is determined to be lacking. An example of the imaging process in that case will be explained with reference to the flowchart in Figure 23.

[0181] When the imaging process begins, the process in step S241 is executed in the same way as the process in step S151 in Figure 16. That is, the main image and 3D information are obtained on the same optical axis.

[0182] In step S242, the 3D information processing unit 149 analyzes the surface irregularities of the object based on the 3D information.

[0183] Each of the processes in steps S243 to S246 is performed in the same manner as the processes in steps S152 to S155 in Figure 16. That is, a YUV image, a reduced image, and metadata are generated.

[0184] In step S247, the 3D information processing unit 149 determines whether the shape of the object is planar or not based on the analysis results from step S242. If it is determined that the shape of the object is planar, the signature control unit 153 proceeds to step S248. That is, the signature control unit 153 controls the process to generate a signature.

[0185] Steps S248 and S249 are performed in the same manner as steps S156 and S157 in Figure 16. That is, a hash value is calculated for information including at least the main image and 3D information, and a signature is generated using the device secret key corresponding to the imaging device 111. In other words, a signature is generated for information including at least the main image and 3D information. Once the process in step S249 is completed, the process proceeds to step S250.

[0186] Furthermore, if it is determined in step S247 that the shape of the subject is not planar, the signature control unit 153 skips the processing in steps S248 and S249 and proceeds to step S250. In other words, the signature control unit 153 controls the process so that no signature is generated.

[0187] In other words, if the 3D information processing unit 149 (plane determination unit) determines that the shape of the object is not plane, the signature generation unit 150 generates a signature. To put it another way, if the 3D information processing unit 149 determines that the shape of the object is plane, the signature generation unit 150 does not generate a signature (it omits the creation of a signature).

[0188] Each of the processes in steps S250 to S252 is executed in the same manner as the processes in steps S158 to S160 in Figure 16. That is, an image file is generated, and that image file is recorded on a removable recording medium or the like by the recording unit 134, or uploaded to the server 112. When the process in step S252 is completed, the imaging process is completed. In this case as well, either the process in step S251 or the process in step S252 may be omitted. Furthermore, if the RAW image and 3D information obtained by the process in step S241, and the metadata generated by the process in step S246 are saved, the processes in the other steps (analysis of 3D information, processing of RAW images, generation of YUV images, generation of reduced images, calculation of hash values, generation of signatures, generation and provision of image files, etc.) may be executed as processes different from the imaging process after the imaging process is completed.

[0189] By performing each process as described above, the imaging device 111 can omit the generation of a signature when the shape of the subject is flat. In other words, the imaging device 111 can avoid generating a signature for images in which the shape of the subject is flat. Consequently, the imaging device 111 can be configured to determine that the image is not authentic when determining the authenticity of the image.

[0190] <2-2-5. Image Processing (AF-S Control)> In single autofocus mode (AF-S), the 3D information sensor 148 may acquire 3D information at the focus lock timing and the main exposure start timing. As described above, for example, when a predetermined operation such as half-pressing the shutter is performed by the user, the control unit 121 performs control to lock the focus on the optical system 141. The focus lock timing is the timing at which the control unit 121 performs this control (the timing at which focus lock is applied). The main exposure start timing is the timing at which the main exposure is started in the image sensor 142. An example of the imaging process flow in this case will be explained with reference to the flowchart in Figure 24.

[0191] When the imaging process begins, in step S271, the 3D information sensor 148 determines whether the operating mode is single autofocus mode (AF-S). If it is determined to be single autofocus mode (AF-S), the process proceeds to step S272. In single autofocus mode, there is a period during which the focal length is fixed before the start of the main exposure, so 3D information is acquired multiple times.

[0192] In step S272, the image sensor 142 generates a RAW image. The 3D information sensor 148 also acquires 3D information from the optical image of the subject at multiple timings, using the same optical axis as the RAW image. In this case, the 3D information sensor 148 acquires 3D information at the focus-fixing timing and the main exposure start timing. Once the processing in step S272 is completed, the process proceeds to step S274.

[0193] Furthermore, if it is determined in step S271 that the camera is not in single autofocus mode (AF-S), for example, in continuous autofocus mode (AF-C) or manual mode, the process proceeds to step S273. Continuous autofocus mode refers to a mode in which the process of focusing on the subject continues while the user performs a predetermined operation (for example, half-pressing the shutter button). Manual mode refers to a mode in which the user manually adjusts the focal length. In these modes, there is no period in which the focal length is fixed before the start of the main exposure, so 3D information is acquired only at the start of the main exposure.

[0194] In step S273, the image sensor 142 generates a RAW image. The 3D information sensor 148 acquires 3D information from the optical image of the subject on the same optical axis as the RAW image. In this case, the 3D information sensor 148 acquires 3D information at the start of the exposure. Once the processing in step S273 is completed, the process proceeds to step S274.

[0195] Each of the processes in steps S274 to S282 is performed in the same manner as the processes in steps S152 to S160 in Figure 16. When the process in step S282 is completed, the imaging process is completed. In this case as well, either the process in step S281 or the process in step S282 may be omitted. Furthermore, if the RAW image and 3D information obtained by the processes in steps S271 to S273, as well as the metadata generated by the process in step S277, are saved, the processes in the other steps (processing of RAW images, generation of YUV images, generation of reduced images, calculation of hash values, generation of signatures, generation and provision of image files, etc.) may be executed as processes separate from the imaging process after the imaging process is completed.

[0196] By performing each process in this manner, the imaging device 111 can acquire 3D information on the same optical axis as the RAW image at the focus-fixing timing and the main exposure start timing in single autofocus mode (AF-S). Therefore, it becomes difficult to perform trick photography, such as acquiring 3D information at the focus-fixing timing and then changing the subject (while keeping the focal length fixed) and taking an image. In other words, it becomes difficult to generate a false image in which the subject differs between the 3D information and the image. This means that the server 112 can more accurately verify the authenticity of the image.

[0197] <2-2-6. Imaging Processing (Reliability Recording)> The reliability of the 3D information may be calculated, and metadata including this reliability information may be stored in the image file. An example of the imaging process in this case will be explained with reference to the flowchart in Figure 25.

[0198] When the imaging process begins, the process in step S301 is executed in the same way as the process in step S241 in Figure 23. That is, the main image and 3D information are obtained on the same optical axis.

[0199] In step S302, the confidence calculation unit 154 analyzes the image obtained by the image sensor 142 in step S301 and estimates the topography of the subject. For example, the confidence calculation unit 154 may determine the features of the image and estimate the topography of the image based on those features. For example, the confidence calculation unit 154 may detect faces, eyes, noses, ears, etc., contained in the image and estimate the topography of each detected part.

[0200] In step S303, the reliability calculation unit 154 compares the analysis result from step S302 (i.e., the shape of the surface in the estimated image) with the shape of the surface indicated by the 3D information, and calculates the reliability of the 3D information based on the comparison result. For example, if the subject (person) is facing the imaging device 111, generally, the nose, which is a protruding part of the face, will be closer to the imaging device 111 (the distance to the imaging device 111 will be shorter) than other parts of the face (e.g., the eyes and ears). Since there is a correlation between the shape of the surface of the subject and the 3D information, the reliability calculation unit 154 uses this correlation to evaluate the image analysis result using the 3D information and calculates the reliability.

[0201] Each of the processes in steps S304 to S306 is performed in the same manner as the processes in steps S243 to S245 in Figure 23. That is, a YUV image and a reduced image are generated.

[0202] In step S307, the metadata addition unit 146 generates metadata including the confidence level calculated in step S303 and adds it to the image. In other words, the metadata addition unit 146 generates metadata other than the confidence level and includes information indicating the confidence level calculated in step S303 in that metadata.

[0203] Each process in steps S308 to S312 is performed in the same manner as each process in steps S248 to S252 in Figure 23. That is, a hash value is calculated and a signature is generated. Then, in step S310, the image file generation unit 151 compresses and encodes the YUV image (main image) to generate a JPEG image (main image). The image file generation unit 151 also generates an image file that stores the main image, 3D information, a reduced image, metadata including information indicating the confidence level calculated in step S303, and the signature. Then, this image file is recorded or uploaded. When the process in step S312 is completed, the imaging process is completed. In this case as well, either the process in step S311 or the process in step S312 may be omitted. Furthermore, if the RAW image and 3D information obtained by the processing in step S301, as well as the metadata (metadata other than confidence level) generated by the processing in step S307 are saved, the processing in the other steps (image analysis, confidence level calculation, processing of RAW images, generation of YUV images, generation of reduced images, calculation of hash values, signature generation, generation and provision of image files, etc.) may be executed as separate processes from the imaging process after the imaging process is completed.

[0204] By performing each process in this manner, the imaging device 111 can calculate the reliability of the 3D information and store information indicating that reliability in the image file. Therefore, the server 112 can perform image analysis based on that reliability. Consequently, the server 112 can more accurately determine the authenticity of the image.

[0205] <2-2-7. Imaging Processing (Shutter Speed ​​Control)> If the image capture shutter speed is slower than a predetermined standard, the 3D information sensor 148 may save 3D information multiple times during the main exposure of the image sensor 142. An example of the imaging process in this case will be explained with reference to the flowchart in Figure 26.

[0206] When the imaging process begins, in step S331, the 3D information sensor 148 determines whether the shutter speed of the image sensor 142 is slower than a predetermined standard. If it is determined that the shutter speed is slower than the predetermined standard, the process proceeds to step S332.

[0207] In step S332, the image sensor 142 generates a RAW image. The 3D information sensor 148 acquires 3D information from the optical image of the subject at multiple timings, using the same optical axis as the RAW image. In this case, the 3D information sensor 148 acquires 3D information multiple times during the main exposure of the image sensor 142. Once the processing in step S332 is complete, the process proceeds to step S334.

[0208] Furthermore, if it is determined in step S331 that the shutter speed is faster than a predetermined standard, the process proceeds to step S333. In step S333, the image sensor 142 generates a RAW image. The 3D information sensor 148 also acquires 3D information from the optical image from the subject on the same optical axis as the RAW image. In this case, the 3D information sensor 148 acquires 3D information at the start timing of the main exposure. When the processing in step S333 is completed, the process proceeds to step S334.

[0209] Each of the processes in steps S334 to S342 is performed in the same manner as the processes in steps S274 to S282 in Figure 24. When the process in step S342 is completed, the imaging process is completed. In this case as well, either the process in step S341 or the process in step S342 may be omitted. Furthermore, if the RAW image and 3D information obtained by the processes in steps S331 to S333, as well as the metadata generated by the process in step S337, are saved, the processes in the other steps (processing of RAW images, generation of YUV images, generation of reduced images, calculation of hash values, generation of signatures, generation and provision of image files, etc.) may be executed as processes separate from the imaging process after the imaging process is completed.

[0210] By performing each process in this manner, the imaging device 111 can detect distance-related information multiple times on the same optical axis as the RAW image and generate 3D information multiple times during the exposure of the image, even when the shutter speed is slower than a predetermined standard.

[0211] Therefore, it becomes difficult to perform trick photography, such as changing the subject during the exposure period. In other words, it becomes difficult to generate false images in which the subject differs between the 3D information and the image. This means that even with a slow shutter speed, the server 112 can more accurately verify the authenticity of the image.

[0212] <2-3. Server Processing> <2-3-1. Verification Process> Next, we will explain each process performed by the server 112, etc. The server 112 verifies the authenticity of the images stored in the image file uploaded by the imaging device 111. An example of this verification process will be explained with reference to the flowchart in Figure 27.

[0213] When the verification process is initiated, the communication unit 221 of the server 112 receives (acquires) the image file transmitted from the imaging device 111 in step S401.

[0214] This image file contains a device-specific ID corresponding to the source imaging device 111 as metadata. In step S402, the signature verification unit 232 obtains the device public key corresponding to that device-specific ID from the device public key database 223. For example, the signature verification unit 232 supplies that device-specific ID to the device public key management unit 231. The device public key management unit 231 accesses the device public key database 223, obtains the device public key associated with that device-specific ID, and supplies it to the signature verification unit 232.

[0215] The signature verification unit 232 then uses its device public key to verify the validity of the signature stored in the image file. Specifically, the signature verification unit 232 confirms that the image and 3D information stored in the image file, as well as the device secret key corresponding to the imaging device 111 that generated them, are used to create the signature.

[0216] For example, the signature verification unit 232 decrypts the signature stored in the image file using the device's public key and obtains a first hash value. The signature verification unit 232 also calculates a second hash value using the main image, 3D information, thumbnail image, and metadata stored in the image file. The signature verification unit 232 then compares the first hash value and the second hash value to confirm whether they match.

[0217] In other words, if the first hash value and the second hash value match, the signature verification unit 232 determines that the signature is valid and that the image or 3D information stored in the image file has not been tampered with. Conversely, if the first hash value and the second hash value do not match, the signature verification unit 232 determines that the signature is invalid and that the image or 3D information stored in the image file has been tampered with.

[0218] In step S403, the signature verification unit 232 determines whether the validity of the signature has been confirmed. If it is determined that the signature is valid as a result of the process in step S402, the process proceeds to step S404.

[0219] In step S404, the image verification processing unit 233 performs an image verification process and verifies the authenticity of the image stored in the image file using the 3D information stored in the image file. In other words, the image verification processing unit 233 verifies the authenticity of the image by comparing the image with 3D information acquired on the same optical axis as the image. When the image verification process in step S404 is completed, the verification process is terminated.

[0220] If, in step S403, it is determined that the validity of the signature could not be confirmed by the processing in step S402 (i.e., the signature is found to be invalid), the process proceeds to step S405. In step S405, the image verification processing unit 233 performs error processing. When the processing in step S405 is completed, the verification process ends. In other words, in this case, the image verification process in step S404 is not executed (the image verification process is omitted (skipped)). In this case, the image file is processed as not being authentic.

[0221] In this way, the signature is used to confirm that the image and 3D information have not been tampered with, so the server 112 can more accurately verify the authenticity of the image using the 3D information. If the signature is determined to be invalid in step S403, the image verification process in step S404 may be executed, and the image verification process may be configured to determine that the image is not authentic.

[0222] <2-3-1-1. Image Verification Process (Automated Verification)> Next, in the image verification process performed in step S404 of Figure 27, any processing may be performed. For example, the server 112 may verify the authenticity of the image. For example, the image verification unit 241 of the image verification processing unit 233 may verify the authenticity of the image by comparing the surface irregularities of the subject in the image detected from the image with the surface irregularities of the subject based on 3D information. In other words, the image verification unit 241 may determine that the image is authentic if the surface irregularities of the subject in the image match the surface irregularities of the subject based on 3D information, and determine that the image is not authentic if they do not match. The server 112 may also present the verification result to the verifier. For example, the support processing unit 242 of the image verification processing unit 233 may present the result of the image authenticity verification to the verifier. An example of the flow of the image verification process in that case will be explained with reference to the flowchart in Figure 28.

[0223] In this case, when the image verification process is started, the image verification unit 241 of the image verification processing unit 233 analyzes the image to determine its features in step S421 and estimates the surface irregularities of the subject based on those features. For example, the image verification unit 241 detects the face, eyes, nose, ears, etc. of a person contained in the image and estimates the surface irregularities of each detected part.

[0224] In step S422, the image verification unit 241 compares the surface topography of the subject in the image estimated in step S421 (analysis result) with the surface topography of the subject indicated by the 3D information. Based on the comparison result, the image verification unit 241 determines the authenticity of the image (whether or not the image is a fake image).

[0225] In step S423, the support processing unit 242 supplies information indicating the comparison result (the determination result of whether the image is a fake image or not) to the terminal device 113 via the communication unit 221 and displays it. The user of the terminal device 113 is the verifier who confirms the authenticity of the image. In other words, the support processing unit 242 presents the comparison result (the determination result of whether the image is a fake image or not) to the verifier.

[0226] Once the process in step S423 is completed, the image verification process ends, and the process returns to Figure 27.

[0227] An example of the flow of the comparison result display processing performed in the terminal device 113 in response to the image verification processing of the server 112 will be explained with reference to the flowchart in Figure 29.

[0228] When the comparison result display process is started, the CPU 301 of the terminal device 113 acquires information indicating the comparison result (the determination result of whether or not the image is a fake image) transmitted from the server 112 via the communication unit 314 in step S431.

[0229] In step S432, the CPU 301 supplies information indicating the comparison result to the output unit 312 for display. The output unit 312 displays an image 501 on a display device, for example, the comparison result (the result of determining whether the main image is a fake image or not), as shown in Figure 30. The content of this image 501 is arbitrary. In the example of Figure 30, image 501 includes the main image 511. Image 501 also includes an image 512 in which the main image and 3D information are superimposed. Image 501 also includes the file name of the main image 511. Furthermore, image 501 includes information such as whether the device public key is valid, whether the signature is valid, and the result of verifying the authenticity of the main image (pass or fail). The verifier (user of terminal device 113) can understand the comparison result (the result of verifying the authenticity of the main image) by viewing this image 501.

[0230] Once the processing in step S432 is complete, the comparison result display process ends.

[0231] By performing each process as described above, the server 112 can verify the authenticity of the image using 3D information detected on the same optical axis as the image. Therefore, the server 112 can determine the authenticity of the image more accurately.

[0232] In the above, it was explained that in step S421 of the image verification process, the image verification unit 241 analyzes the image to determine its features and estimates the surface irregularities based on those features. However, a screen nail (reduced image) may be used instead of the image. That is, the image verification unit 241 may analyze the screen nail (reduced image) to determine its features and estimate the surface irregularities based on those features.

[0233] <2-3-1-2. Image Verification Process (Comparison and Display)> Furthermore, in step S404 of Figure 27, the server 112 may present to the verifier the result of comparing the surface features of the image with the surface features of the same object based on 3D information, as information to help the verifier confirm whether or not the image is a fake. An example of the image verification process in this case will be explained with reference to the flowchart in Figure 31.

[0234] In this case, when the image verification process is started, steps S451 and S452 are executed in the same way as steps S421 and S422 in Figure 28. That is, the topography of the subject in the image is estimated, and the estimated topography of the subject in the image (analysis result) is compared with the topography of the subject as shown by the 3D information. However, in this case, the authenticity of the image (whether the image is a fake or not) is not determined.

[0235] In step S453, the image verification unit 241 generates auxiliary lines to indicate the matching of the uneven surfaces in the comparison in step S452.

[0236] In step S454, the support processing unit 242 supplies information indicating the comparison result to the terminal device 113 via the communication unit 221 and displays it. For example, the support processing unit 242 supplies the main image, 3D information, and information such as auxiliary lines generated in step S453 to the terminal device 113 and displays it as reference information for confirming the authenticity of the image.

[0237] Once the process in step S454 is completed, the image verification process ends, and the process returns to Figure 27.

[0238] An example of the flow of the comparison result display processing performed in the terminal device 113 in response to the image verification processing of the server 112 will be explained with reference to the flowchart in Figure 32.

[0239] When the comparison result display process is started, the CPU 301 of the terminal device 113 acquires information indicating the comparison result (for example, the main image, 3D information, and auxiliary lines) transmitted from the server 112 via the communication unit 314 in step S461.

[0240] In step S462, the CPU 301 supplies information regarding the comparison result to the output unit 312. The output unit 312 displays the image and 3D information on a display device in a way that allows the viewer to compare them. The output unit 312 may also display auxiliary lines on the display device. For example, the output unit 312 displays a confirmation screen 521 on the display device, as shown in Figure 33.

[0241] This confirmation screen 521 is a screen for the verifier to confirm the authenticity of the image. The content of this confirmation screen 521 is arbitrary. For example, as shown in Figure 33, an OK button 534 and an NG button 535 may be displayed on this confirmation screen 521. For example, if the user operates the input unit 311 and presses the OK button 534, the authenticity of the image is affirmed. That is, the verifier determines that the image is not a fake image. Also, if the user operates the input unit 311 and presses the NG button 535, the authenticity of the image is denied. That is, the verifier determines that the image is a fake image.

[0242] Furthermore, as shown in Figure 33, the confirmation screen 521 may display the main image 531, the superimposed image 532 of the main image and 3D information, and the superimposed image 533 of the main image and auxiliary lines as reference information for the confirmer to perform such confirmation. For example, the main image 531 allows the confirmer to visually understand what kind of image the main image to be confirmed is. The superimposed image 532 allows the confirmer to visually understand how the surface irregularities of the main image and the 3D information match, or do not match. In addition, the auxiliary lines in the superimposed image 533 allow the confirmer to visually understand which parts match.

[0243] Note that the layout of this confirmation screen 521 is arbitrary. For example, this image and 3D information may be displayed side by side, superimposed, or alternately. Also, the UI that prompts the determination of authenticity (such as the OK button 534 and NG button 535) may be omitted. It may simply display the comparison result. Also, the auxiliary lines may be displayed not as simple lines but so that the distance can be gently recognized.

[0244] When the process of step S462 ends, the comparison result display process ends.

[0245] By executing each process as described above, the server 112 compares the concavo-convex state between this image and the 3D information detected on the same optical axis as this image, and presents the comparison result to the verifier, thereby enabling the verifier to confirm the authenticity of the image. Therefore, the server 112 can more accurately determine the authenticity of the image.

[0246] <2-3-1-3. Image Confirmation Process (Display)> Note that in step S403 of FIG. 27, the server 112 may present the image and 3D information to be compared to the verifier without comparing the concavo-convex state of this image and the 3D information. An example of the flow of the image confirmation process in that case will be described with reference to the flowchart of FIG. 34.

[0247] In this case, when the image confirmation process starts, in step S481, the support processing unit 242 of the server 112 supplies the image and 3D information to be compared to the terminal device 113 for display. The support processing unit 242 presents this information to the verifier as reference information for the user of the terminal device 113, who is the verifier, to confirm the authenticity of the image.

[0248] When the process of step S481 ends, the image confirmation process ends, and the process returns to FIG. 27.

[0249] An example of the flow of the comparison result display process executed in the terminal device 113 in response to the image confirmation process of such a server 112 will be described with reference to the flowchart of FIG. 35.

[0250] When the comparison result display process is started, the CPU 301 of the terminal device 113 acquires the original image and 3D information transmitted from the server 112 via the communication unit 314 in step S491.

[0251] In step S492, the CPU 301 supplies the original image and 3D information to the output unit 312. The output unit 312 displays the original image and 3D information on the display device in a state where the verifier can compare them. For example, the output unit 312 displays a confirmation screen 521 as shown in FIG. 33 on the display device. However, in this case, the display of the superimposed image 533 is omitted.

[0252] From the original image 531 of this confirmation screen 521, the verifier can visually grasp what kind of image the original image to be confirmed is. Also, from the superimposed image 532 of the confirmation screen 521, the verifier can visually grasp how the concavity and convexity match or do not match between the original image and the 3D information. The verifier determines the authenticity of the original image by referring to these images and operates the OK button 534 or the NG button 535. In this way, the authenticity of the original image is confirmed by the verifier.

[0253] When the process of step S492 ends, the comparison result display process ends.

[0254] By executing each process as described above, the server 112 can present the original image and the 3D information detected on the same optical axis as the original image to the verifier, thereby allowing the verifier to confirm the authenticity of the image. Therefore, the server 112 can more accurately determine the authenticity of the image.

[0255] <2-3-2. Confirmation Process (Metadata Update)> If the authenticity of an image cannot be verified, the metadata (such as rotation and rating) may be updated and the verification process performed again. An example of the verification process in this case is explained with reference to the flowchart in Figure 36.

[0256] Once the verification process begins, steps S511 to S513 are executed in the same manner as steps S401 to S403 in Figure 27. That is, the image file is acquired, and the signature and the authenticity of the image are verified.

[0257] In step S514, the confirmation control unit 234 determines whether the image and the 3D information match. If it is determined that they do not match, the process proceeds to step S515.

[0258] In step S515, the verification control unit 234 updates the metadata (such as image rotation and rating) and returns the process to step S512. That is, the verification control unit 234 updates the metadata and performs signature verification and image authenticity verification again.

[0259] Then, if it is determined in step S514 that the image and the 3D information match, the verification process ends.

[0260] By performing each process as described above, the server 112 can more accurately determine the authenticity of an image even if the image has been rotated or otherwise processed.

[0261] <2-3-3. Verification Process (Public Key Invalidation Determination)> Furthermore, the verification process may include a check to determine whether or not the device's public key is invalid. An example of the verification process in this case will be explained with reference to the flowchart in Figure 37.

[0262] Once the verification process begins, the process in step S531 is executed in the same way as the process in step S401 in Figure 27. That is, an image file is acquired.

[0263] In step S532, the device public key management unit 231 searches the device public key database 223 for and retrieves the device public key corresponding to the device-specific ID contained in the image file.

[0264] In step S533, the device public key management unit 231 determines whether the device public key is valid or not. If the device public key is not invalidated in the device public key database 223, that is, if the current date is before the invalidation date of the device public key, the process proceeds to step S534.

[0265] In this case, steps S534 and S535 are performed in the same way as steps S402 and S403 in Figure 27. That is, the signature is verified and the authenticity of the image is verified. When step S535 is completed, the verification process is finished.

[0266] Furthermore, if it is determined in step S533 that the device public key is invalid, that is, if the current date is after the invalidation date of the device public key, the process proceeds to step S536. In step S536, the image verification processing unit 233 performs error processing. When the processing in step S536 is completed, the verification process is completed. In other words, in this case, the signature verification and image authenticity verification processes are omitted (skipped).

[0267] By performing each process in this manner, it is possible to avoid verifying signatures using invalid device public keys. Therefore, server 112 can more accurately determine the authenticity of the image.

[0268] <2-3-4. Verification Process (Confirmation of Same Optical Axis)> Furthermore, in the above explanation, it has been explained that the uploading of image files to the server 112 is always performed from the imaging device 111. In other words, in this case, the 3D information contained in the image file is always obtained using the same optical axis as the main image.

[0269] The upload of the image file to this server 112 may also be performed from other than the imaging device 111. That is, the 3D information included in the image file may not be limited to that detected on the same optical axis as this image. And only when the 3D information is detected on the same optical axis as this image, signature confirmation and confirmation of the authenticity of the image may be performed. An example of the flow of the confirmation process in that case will be described with reference to the flowchart of FIG. 38.

[0270] When the confirmation process is started, the process of step S551 is executed in the same manner as the process of step S401 in FIG. 27. That is, an image file is acquired.

[0271] In step S552, the optical axis determination unit 235 determines whether the present image and the 3D information are obtained on the same optical axis based on the metadata included in the image file.

[0272] For example, when flag information of a value indicating that the present image and the 3D information are obtained on the same optical axis is stored in the image file, the optical axis determination unit 235 determines that the present image and the 3D information are obtained on the same optical axis. Also, when the device name, model name, or identification information of a device in which the present image and the 3D information are always obtained on the same optical axis is stored in the image file, the optical axis determination unit 235 determines that the present image and the 3D information are obtained on the same optical axis. In other words, when those pieces of information are not stored in the image file, the optical axis determination unit 235 determines that the present image and the 3D information are obtained on different optical axes.

[0273] When it is determined that the present image and the 3D information are obtained on the same optical axis, the process proceeds to step S553. In this case, each process from step S553 to step S556 is executed in the same manner as each process from step S402 to step S405 in FIG. 27. That is, the validity of the signature is confirmed, and depending on the result of the confirmation, the authenticity of the image or error processing is performed. When the process of step S555 or step S556 ends, the confirmation process ends.

[0274] Furthermore, if it is determined in step S552 that the image and 3D information were obtained on different optical axes, the process proceeds to step S556. In this case, the process in step S556 is executed in the same way as the process in step S405 in Figure 27. That is, error handling is performed. When the process in step S556 is completed, the verification process is completed. In this case, the signature verification and the verification of the authenticity of the image are omitted (skipped). In this case, the image file is processed as not being authentic.

[0275] By performing each process in this manner, the server 112 can perform signature verification and image authenticity verification only if the 3D information is detected on the same optical axis as the main image. Therefore, the server 112 can more accurately determine the authenticity of the image. If the signature is determined to be invalid in step S554, the image verification process in step S555 may be executed, and the image may be determined to be inauthentic in that process.

[0276] <2-3-5. Verification Process (Confidence Level Determination)> Furthermore, based on information indicating the reliability of the 3D information stored as metadata in the image file, signature verification and image authenticity verification may be performed only if the 3D information is deemed sufficiently reliable. An example of the verification process in this case will be explained with reference to the flowchart in Figure 39.

[0277] Once the verification process begins, the process in step S571 is executed in the same way as the process in step S401 in Figure 27. That is, an image file is acquired.

[0278] In step S572, the confidence determination unit 236 analyzes the image stored in the image file and extracts segments from which the surface irregularities of the subject can be estimated.

[0279] In step S573, the reliability determination unit 236 refers to information indicating the reliability of the 3D information stored as metadata in the image file and determines whether the 3D information for the segment extracted in step S572 is sufficiently reliable. In other words, the reliability determination unit 236 determines whether the 3D information is reliable based on the reliability information indicating the reliability of the 3D information, which is stored as metadata about the image in the image file that stores the image and the 3D information.

[0280] If there is a sufficient amount of 3D information for that segment with a reliability higher than a predetermined standard (more than the predetermined standard), the reliability determination unit 236 determines that the 3D information for that segment is sufficiently reliable. Conversely, if there is a sufficient amount of 3D information for that segment with a reliability higher than a predetermined standard, the reliability determination unit 236 determines that the 3D information for that segment is unreliable.

[0281] If the 3D information for the extracted segment is deemed sufficiently reliable, the process proceeds to step S574. In this case, steps S574 through S577 are performed in the same manner as steps S402 through S405 in Figure 27. That is, the validity of the signature is verified, and depending on the result of that verification, the authenticity of the image is verified or error handling is performed. The verification process ends when step S576 or step S577 is completed.

[0282] Furthermore, if it is determined in step S573 that the 3D information for the extracted segment is unreliable, the process proceeds to step S577. In this case, the process in step S577 is executed in the same way as the process in step S405 in Figure 27. That is, error handling is performed. When the process in step S577 is completed, the verification process is completed. In this case, the signature verification and image authenticity verification processes are omitted (skipped). In this case, the image file is processed as if it is not authentic.

[0283] By performing each process in this manner, the server 112 can perform signature verification and image authenticity verification only if the 3D information is sufficiently reliable. Therefore, the server 112 can more accurately determine the authenticity of the image. If the signature is determined to be invalid in step S575, the image verification process in step S576 may be executed, and the image may be determined to be inauthentic in that process.

[0284] <2-3-6. Verification Process (Confidence Calculation)> Server 112 may calculate the reliability of the 3D information and present the calculated reliability to the verifier. An example of the verification process in this case will be explained with reference to the flowchart in Figure 40.

[0285] Once the verification process begins, steps S591 to S593 are executed in the same manner as steps S401 to S403 in Figure 27. That is, an image file is acquired, the validity of the signature is verified, and it is determined whether or not its validity has been verified. In step S593, if it is determined that the signature is valid based on the process in step S592, the process proceeds to step S594.

[0286] In step S594, the reliability calculation unit 237 calculates the reliability of the 3D information stored in the image file based on the metadata (camera parameters related to the image) stored in the image file. For example, the reliability calculation unit 237 calculates the reliability based on camera parameters such as the number of effective pixels, F-number, and focal length of the sensor unit 161 (image sensor 142 and 3D information sensor 148) of the imaging device 111, which are stored in the image file. This method of calculating reliability is arbitrary.

[0287] In step S595, the image verification processing unit 233 performs image verification processing. This process is basically performed in the same way as in step S404 in Figure 27. However, in this case, the image verification processing unit 233 performs image verification processing using the confidence level of the 3D information calculated in step S594. For example, the support processing unit 242 may present information indicating the confidence level of the 3D information to the verifier along with the comparison result of the surface irregularities of the image and the 3D information. Alternatively, the image verification unit 241 may use the confidence level of the 3D information to determine the authenticity of the image. When the processing in step S595 is completed, the verification process is completed.

[0288] Furthermore, if, in step S593, it is determined that the signature could not be confirmed to be valid (i.e., the signature was confirmed to be invalid) based on the processing in step S592, the process proceeds to step S596. In step S596, the image verification processing unit 233 performs error processing. Once the processing in step S596 is completed, the verification process ends. In other words, in this case, the calculation of confidence level in step S594 and the image verification processing in step S595 are not performed (these processes are omitted (skipped)). In this case, the image file is treated as not being authentic.

[0289] By executing each process in this manner, the server 112 can, for example, calculate the reliability of the 3D information and present that reliability to the verifier. This allows the verifier to understand the reliability of the 3D information. For example, if the server 112 verifies the authenticity of an image and presents the verification result to the verifier, the verifier can understand the likelihood of that verification result. Also, when the verifier verifies the authenticity of an image, the verifier can more accurately determine the authenticity of the image based on the reliability of the 3D information. Furthermore, the server 112 can, for example, calculate the reliability of the 3D information and use that reliability to determine the authenticity of an image. This allows the server 112 to more accurately determine the authenticity of an image. Note that if the signature is determined to be invalid in step S593, the reliability calculation process in step S594 and the image verification process in step S595 may be executed, and the image verification process may determine that the image is not authentic.

[0290] <2-3-7. Verification Process (Shutter Speed ​​Determination)> As described above in <2-2-3. Imaging Processing>, etc., when the shutter speed is slow, that is, when the exposure period is long, trick photography can be performed by changing the subject during exposure. In other words, it is possible to generate a false image in which the subject differs between the 3D information and the image. For example, if the shutter speed is set to 10 seconds in a dark room, the imaging device 111 is pointed at a person until just before shooting, and then the imaging device 111 is pointed at the composite image monitor after the exposure has started, an image of a different subject from the subject shown in the 3D information can be obtained.

[0291] Therefore, the shutter speed during image generation may be determined, and signature verification and image authenticity verification may only be performed if the shutter speed is sufficiently fast. An example of the verification process in this case will be explained with reference to the flowchart in Figure 41.

[0292] Once the verification process begins, the process in step S611 is executed in the same way as the process in step S401 in Figure 27. That is, an image file is acquired.

[0293] In step S612, the shutter speed determination unit 238 refers to information indicating the shutter speed during image capture, which is stored in the image file as a camera parameter related to the image, and determines whether the shutter speed during image generation is sufficiently fast. If the shutter speed indicated in that information is faster than a predetermined standard, the shutter speed determination unit 238 determines that the shutter speed during image generation is sufficiently fast. Conversely, if the shutter speed indicated in that information is slower than a predetermined standard, the shutter speed determination unit 238 determines that the shutter speed during image generation is not sufficiently fast.

[0294] If it is determined that the shutter speed during image generation is sufficiently fast (faster than a predetermined standard), the process proceeds to step S613. In this case, each of the processes in steps S613 to S616 is executed in the same manner as the processes in steps S402 to S405 in Figure 27. That is, the validity of the signature is checked, and depending on the result of that check, the authenticity of the image is checked or error handling is performed. When the process in step S615 or step S616 is completed, the verification process ends.

[0295] Furthermore, if it is determined in step S612 that the shutter speed during image generation is not sufficiently fast (slower than a predetermined standard), the process proceeds to step S616. In this case, the process in step S616 is executed in the same way as the process in step S405 in Figure 27. That is, error handling is performed. When the process in step S616 is completed, the verification process is completed. In this case, the signature verification and image authenticity verification processes are omitted (skipped). In this case, the image file is processed as if it does not contain the authenticity of the image.

[0296] By executing each process in this manner, the server 112 can perform signature verification and image authenticity verification only if the shutter speed during image generation is sufficiently fast. In other words, the server 112 can perform signature verification and image authenticity verification only if the trick photography described above is difficult and there is a high probability that the image is not a fake image. Therefore, the server 112 can determine the authenticity of the image more accurately. If the signature is determined to be invalid in step S614, the image verification process in step S615 may be executed, and the image may be determined to be inauthentic in that image verification process.

[0297] <2-3-8. Verification Process (JPEG Image Signature)> For example, if both a RAW image and a JPEG image are stored in an image file as the main image, and only the signature of the JPEG image is stored in that image file, the server 112 may determine whether the RAW image and the JPEG image match. If the RAW image and the JPEG image match, the server 112 may also determine that the signature of the RAW image exists. In other words, in this case, the server 112 determines that the RAW image has not been tampered with based on the determination that the JPEG image has not been tampered with based on the signature of the JPEG image. An example of the verification process in this case will be explained with reference to the flowchart in Figure 42.

[0298] Once the verification process begins, the process in step S631 is executed in the same way as the process in step S401 in Figure 27. That is, an image file is acquired.

[0299] In step S632, the development processing unit 239 determines whether the image file contains both a RAW image and a JPEG image as the main image, and whether only the JPEG image has a signature. If it is determined that the image file contains both a RAW image and a JPEG image as the main image, that the signature of the JPEG image is stored in the image file, and that the signature of the RAW image is not stored in the image file, the process proceeds to step S633.

[0300] In step S633, the development processing unit 239 generates a JPEG image using the RAW image. At that time, the development processing unit 239 generates the JPEG image from the RAW image in the same manner as the imaging device 111.

[0301] In step S634, the development processing unit 239 compares the JPEG image generated in step S633 with the JPEG image stored in the image file and determines whether the two match. If it is determined that these JPEG images match, the process proceeds to step S635. In other words, if the RAW image and JPEG image are stored as the main image in the image file, the signature of the JPEG image is stored in the image file, the signature of the RAW image is not stored in the image file, and the JPEG image generated from the RAW image matches the JPEG image stored in the image file, the process proceeds to step S635.

[0302] Furthermore, in step S632, if it is determined that the image file acquired in step S631 does not contain a RAW image or a JPEG image as the main image (i.e., the main image is either a RAW image or a JPEG image), the process proceeds to step S635. Also, in step S632, if it is determined that the image file does not contain a JPEG image signature, the process proceeds to step S635. Also, in step S632, if it is determined that the image file contains a RAW image signature, the process proceeds to step S635.

[0303] In this case, each process in steps S635 to S638 is executed in the same way as each process in steps S402 to S405 in Figure 27. That is, the validity of the signature is checked, and depending on the result of that check, the authenticity of the image is checked or error handling is performed. In other words, if the signature of the JPEG image stored in the image file is checked, the server 112 determines that not only the JPEG image but also the RAW image has not been tampered with. Then, the authenticity of the image (RAW image and JPEG image) is checked through the image check process. When the process in step S637 or step S638 is completed, the check process is completed.

[0304] Furthermore, if, in step S634, it is determined that the JPEG image generated from the RAW image contained in the image file does not match the JPEG image stored in that image file, the process proceeds to step S638. In this case, the process in step S638 is executed in the same way as the process in step S405 in Figure 27. That is, error handling is performed. When the process in step S638 is completed, the verification process is completed. In this case, the signature verification and the verification of the authenticity of the image are omitted (skipped). In this case, the image file is processed as not being authentic.

[0305] By executing each process in this manner, the server 112 can determine that both the RAW image and the JPEG image are stored in the image file as the main image, only the signature of the JPEG image is stored in that image file, and if the RAW image and the JPEG image match, the server can treat the RAW image as unaltered based on the determination that the JPEG image has not been tampered with due to the signature of the JPEG image. If the signature is determined to be invalid in step S636, the image verification process in step S637 may be executed, and the image verification process may be configured to determine that the image is not authentic.

[0306] <3.Reduced image signature> In the above explanation, the signature of the main image is stored in the image file, but the signature of a thumbnail image may be stored instead. A typical example of the image file structure in that case is shown in Figure 43.

[0307] As shown in Figure 43, in this case, a signature (reduced image signature 552) is generated using the reduced image 434, standard metadata 441, and additional metadata 442 (device-specific ID 411, 3D information 432, and shooting information 451, etc.), excluding the main image 431, and this reduced image signature 552 is stored in the image file 440 as additional metadata 442.

[0308] In other words, since the reduced image signature 552 does not include the main image 431, the reduced image signature 552 has less data than signature 452 (Figure 19) which includes the main image 431. Therefore, the reduced image signature 552 can be generated faster with less overhead than signature 452.

[0309] For example, in continuous shooting mode, where multiple images are taken while the imaging operation is ongoing, if it takes time to generate the signature, the image file generation may not keep up with the generation of the captured images. For instance, suppose that in continuous shooting mode, 10 images are taken per second, and a 20MB main image is obtained each time. If it takes 0.9 seconds to generate the signature for that 20MB main image, the image file generation speed will not keep up with the imaging speed, and the imaging pace will drop to about once per second.

[0310] Therefore, when generating the main image in continuous shooting mode, this reduced image signature 552 may be generated instead of signature 452. An example of the imaging process flow performed by the imaging device 111 in that case will be explained with reference to the flowchart in Figure 44.

[0311] Once the imaging process begins, steps S701 to S705 are executed in the same manner as steps S151 to S155 in Figure 16. That is, the RAW image and 3D information are obtained on the same optical axis, the RAW image is corrected, and the YUV image, reduced image, metadata, etc., are generated.

[0312] In step S706, the signature control unit 153 obtains information on the imaging mode of the image sensor 142 from the control unit 121 and determines whether the imaging mode is continuous shooting mode. If it is determined to be continuous shooting mode, the process proceeds to step S707.

[0313] In step S707, the hash processing unit 147 calculates a hash value using the reduced image, standard metadata, and additional metadata (device-specific ID, 3D information, and shooting information, etc.) as described above. Then, in step S708, the signature generation unit 150 encrypts the hash value using the device secret key corresponding to the imaging device 111 read from the storage unit 131, and generates a reduced image signature. The signature generation unit 150 may also generate this reduced image signature by encrypting the hash value using a common key instead of the device secret key. When the processing in step S708 is completed, the process proceeds to step S711.

[0314] Furthermore, if it is determined in step S706 that the imaging mode of the image sensor 142 is not continuous shooting mode (it is single-shot mode, where one image is taken for each imaging operation), the process proceeds to step S709. In this case, the processes in steps S709 and S710 are executed in the same way as the processes in steps S156 and S157 in Figure 16. That is, a hash value is calculated using the image, and a signature including the image is generated. When the process in step S710 is completed, the process proceeds to step S711.

[0315] Each of the processes in steps S711 to S713 is executed in the same manner as the processes in steps S158 to S160 in Figure 16. That is, an image file containing the reduced image signature, etc., is generated, and that image file is recorded or uploaded to the server 112. When the process in step S713 is completed, the imaging process is completed. In this case as well, either the process in step S712 or the process in step S713 may be omitted. Furthermore, if the RAW image and 3D information obtained by the process in step S701, and the metadata generated by the process in step S705 are saved, the processes in the other steps (processing of the RAW image, generation of the YUV image, generation of the reduced image, calculation of the hash value, generation of the signature or reduced image signature, generation and provision of the image file, etc.) may be executed as processes different from the imaging process after the imaging process is completed.

[0316] By performing each process in this manner, the imaging device 111 can generate image files without delaying image capture, even when the image sensor 142 is operating in continuous shooting mode.

[0317] An example of the verification process performed by server 112 on the image file generated in this way, i.e., the image file that stores the thumbnail signature, will be explained with reference to the flowchart in Figure 45.

[0318] When the verification process begins, the process in step S751 is executed in the same way as the process in step S401 in Figure 27. That is, the image file transmitted from the imaging device 111 is acquired.

[0319] In step S752, the signature verification unit 232 reduces the size of the main image contained in the image file to generate a reduced image. This method for generating the reduced image is performed in the same way as when the imaging device 111 generates a reduced image. This method for generating the reduced image may be shared in advance between the imaging device 111 and the server 112, or information indicating the generation method may be stored in the image file as metadata, etc.

[0320] In step S753, the signature verification unit 232 accesses the device public key database 223 via the device public key management unit 231 and obtains the device public key corresponding to the device-specific ID contained in the image file. Then, the signature verification unit 232 uses the reduced image generated in step S752 and its device public key to verify the validity of the reduced image signature.

[0321] For example, the signature verification unit 232 calculates a first hash value using the reduced image generated in step S752, as well as the 3D information and metadata stored in the image file. The signature verification unit 232 also decrypts the reduced image signature stored in the image file using the device public key corresponding to the device-specific ID contained in the image file, and calculates a second hash value. The signature verification unit 232 then determines whether the first hash value and the second hash value match.

[0322] If the first hash value and the second hash value match, the signature verification unit 232 determines that the reduced image signature is valid and that the main image, 3D information, and other information stored in the image file have not been tampered with. Conversely, if the first hash value and the second hash value do not match, the signature verification unit 232 determines that the reduced image signature is invalid and that the main image, 3D information, and other information stored in the image file have been tampered with.

[0323] In step S754, the signature verification unit 232 determines whether the validity of the reduced image signature has been confirmed. If the process in step S753 determines that the reduced image signature is valid, the process proceeds to step S755. In this case, the process in step S755 is executed in the same way as the process in step S404 in Figure 27. That is, the image verification process is executed, and the authenticity of the image is confirmed. When the process in step S755 is completed, the verification process is completed.

[0324] If, in step S754, it is determined that the reduced image signature is not valid (i.e., invalid) based on the processing in step S753, the process proceeds to step S756. In this case, the processing in step S756 is performed in the same way as the processing in step S405 in Figure 27. That is, error handling is performed. When the processing in step S756 is completed, the verification process is completed. In this case, the verification process for the authenticity of the image is omitted (skipped). In this case, the image file is processed as not being authentic.

[0325] By performing each process as described above, the server 112 can detect tampering with the image or 3D information using the reduced image signature. Therefore, the server 112 can more accurately verify the authenticity of the image while suppressing an increase in load. If the signature is determined to be invalid in step S754, the image verification process in step S755 may be executed, and the image verification process may be configured to determine that the image is not authentic.

[0326] Although the above explanation describes applying the reduced image signature to the continuous shooting mode, the reduced image signature can be applied to any operating mode. For example, the reduced image signature may be applied to the single-shot mode. It may also be applied to the video mode, which obtains moving images through imaging.

[0327] <4. Reflection of image processing results in 3D information> In the imaging device 111, when processing or editing an image generated by imaging performed by the image sensor 142, the details of the processing and editing may also be reflected in the 3D information generated by the 3D information sensor 148. For example, when cropping an image, the imaging device 111 may perform the same cropping on the 3D information corresponding to that image. Furthermore, information indicating the details of the processing and editing may be stored as metadata in the image file.

[0328] In such cases, signatures (also called post-processing signatures) of various information stored in the image file, including each edited main image and 3D information, may be generated and stored in the image file. In this case, the post-processing signature may be generated using the server secret key, which is the device secret key corresponding to server 112. An example of the imaging process flow in this case will be explained with reference to the flowchart in Figure 46.

[0329] Once the imaging process begins, steps S801 to S807 are executed in the same manner as steps S151 to S157 in Figure 16. That is, the RAW image and 3D information are obtained on the same optical axis, correction processing is performed on the RAW image, a YUV image (or JPEG image), a reduced image, metadata, etc. are generated, their hash values ​​are calculated, and a signature is generated using these hash values ​​and the instrument secret key.

[0330] In step S808, the YUV processing unit 144 performs processing and editing on the image. The 3D information processing unit 149 performs similar processing and editing on the 3D information. For example, the YUV processing unit 144 crops a portion of image 431 in Figure 18 and extracts image 601 in Figure 47. In this case, the 3D information processing unit 149 crops the 3D information 432 in Figure 18 to the same region as the cropped region of image 431 and extracts 3D information 602 in Figure 47. In other words, image 601 and 3D information 602 are extracted from the same region. Therefore, as shown in the superimposed image 603 in Figure 47, the topography of the subject estimated from image 601 matches the topography shown in 3D information 602.

[0331] In step S809, the hash processing unit 147 calculates a hash value (also referred to as the processed hash value) using the processed image and 3D image etc. that were processed and edited in step S808.

[0332] In step S810, the signature generation unit 150 encrypts the processed and edited hash value using the server secret key and generates the processed and edited signature.

[0333] Each of the processes in steps S811 to S813 is executed in the same manner as the processes in steps S158 to S160 in Figure 16. That is, an image file is generated, and the processed and edited main image and 3D image, reduced image, metadata, processed and edited signature, etc. are stored in that image file, and that image file is recorded or uploaded to server 112. When the process in step S813 is completed, the imaging process is completed. In this case as well, either the process in step S812 or the process in step S813 may be omitted. Also, if the RAW image and 3D information obtained by the process in step S801, and the metadata generated by the process in step S805 are saved, the processes in the other steps (setting the signature execution mode, processing the RAW image, generating a YUV image, generating a reduced image, calculating a hash value, generating a signature, processing and editing, calculating a processed and edited hash value, generating a processed and edited signature, generating and providing an image file, etc.) may be executed as processes different from the imaging process after the imaging process is completed.

[0334] By performing each process in this manner, the imaging device 111 can store the processed and edited image, 3D information, and post-processing signature in the image file. Therefore, the server 112 can verify the authenticity of the processed and edited image as well.

[0335] Furthermore, the post-processed signature generated using the server's private key may or may not include the entire image before processing or editing as the subject of the signature.

[0336] Furthermore, if an image or 3D information is processed or edited multiple times, the post-processing signature may be generated using the most recently processed or edited image as the signature target, or a post-processing signature may be generated each time processing or editing is performed.

[0337] Furthermore, when processing and editing are performed to combine multiple images, the imaging device 111 may store the 3D information corresponding to each image and an image representing the composite interface of each image in an image file. Alternatively, the imaging device 111 may combine the 3D information corresponding to each image in the same way as the image itself.

[0338] In this case, server 112 performs the verification process as described with reference to the flowchart in Figure 27, verifying the signature and the authenticity of the image. However, in this case, server 112 uses the server's private key to verify the signature after processing and editing.

[0339] <5. Application Examples> <5-1. Server Verification> The comparison of images and 3D information by server 112 (comparison of surface features) may be performed using feature points such as faces or people. Alternatively, this comparison may be performed using AI (Artificial Intelligence).

[0340] Furthermore, for example, server 112 may use 3D information to verify that features that should be relatively close in the image are close, and features that should be far away are far away. The features to be verified can be any features. For example, they do not have to be a person's face or nose. For example, AI may be applied to this verification, and the verification may be performed without knowing what the specific features are.

[0341] <5-2. Communications> The exchange of image files between the imaging device 111 and the server 112 may be performed via communication, or via a removable recording medium such as an SD card (registered trademark).

[0342] <5-3. Image Format> The signature may include all information within the image file, including the original image with its original pixel count, or it may exclude the image from the signature. Furthermore, both a signature including the image and a signature excluding the image (e.g., a thumbnail signature) may be generated and stored in the image file.

[0343] Furthermore, images, 3D information, and other supplementary information may be signed in a single file, or they may be kept separate, with instructions on how to combine them being specified, and then signed after being arranged in that manner.

[0344] Furthermore, the signature may be stored in an image file, or it may be stored in a file separate from the image file.

[0345] Furthermore, the signature can be either an electronic signature or a hash using an RSA (Rivest-Shamir-Adleman cryptosystem) private key. These signatures can be generated from the entire data, or they can be applied to a hash value calculated by performing a high-speed hash generation process such as Sha256 on the data.

[0346] This image may be stored as a single image file, or multiple images may be stored within a single image file. The same applies to thumbnail images.

[0347] The format of the image stored in the image file is arbitrary and does not have to be a JPEG image. For example, the image may be a RAW image, a HEIF (High Efficiency Image File Format) image, or a PNG (Portable Network Graphics) image. Of course, other formats are also acceptable.

[0348] A shared private key for each device (also called a device-specific private key) may be established, and the image signature may be performed using that device-specific private key. Server 112 may then verify the signature using a shared public key for each device (also called a device-specific public key).

[0349] The signature on the image may be replaced with a hash value of the image, such as SHA, encrypted using the server's public key. The server 112 may then decrypt the signature using the server's private key and verify its authenticity. Alternatively, the imaging device 111 and the server 112 may apply a shared key. That is, the imaging device 111 may generate a signature using that shared key, and the server 112 may use that shared key to verify the signature.

[0350] <6. Addendum> Furthermore, any combination of the above-described methods may be applied, provided that no contradiction arises. Additionally, the above-described methods may be applied in combination with any other methods not mentioned above.

[0351] <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0352] Figure 48 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0353] In the computer 900 shown in Figure 48, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0354] An input / output interface 910 is also connected to the bus 904. The input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0355] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives a removable recording medium 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0356] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also stores data necessary for the CPU 901 to perform various processes as appropriate.

[0357] The program executed by the computer can be recorded and applied, for example, on a removable recording medium 921 such as a packaged medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable recording medium 921 into the drive 915.

[0358] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0359] Additionally, this program can be pre-installed on ROM902 or memory unit913.

[0360] <Applicable subjects of this technology> This technology can be applied to any image encoding and decoding scheme.

[0361] Furthermore, this technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0362] Furthermore, this technology can also be implemented as part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0363] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0364] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0365] <Other> The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0366] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0367] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0368] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0369] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0370] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0371] Furthermore, this technology can also be configured as follows. (1) An image acquisition unit that acquires an image of the subject by capturing an optical image from the subject, A 3D information acquisition unit that acquires 3D information from the optical image on the same optical axis as the aforementioned image, A signature generation unit that generates signatures for the aforementioned image and the aforementioned 3D information. An image processing device equipped with the following features. (2) The 3D information includes distance-related information for multiple locations within the image, or information generated based on the distance-related information. (1) The image processing apparatus described above. (3) The 3D information acquisition unit acquires the 3D information using a phase difference method. The image processing apparatus described in (1) or (2). (4) The 3D information acquisition unit acquires the 3D information at the exposure start timing when the main exposure is started in the image acquisition unit. (1) to (3) the image processing apparatus described in any of these three terms. (5) In single autofocus mode, the 3D information acquisition unit acquires the 3D information at the focus locking timing and the exposure start timing, which are controlled to fix the focus. (4) The image processing apparatus described above. (6) The system further comprises an image file generation unit that generates an image file containing the image, the 3D information, metadata including information indicating that the 3D information was acquired on the same optical axis as the image, and the signature. (1) The image processing apparatus described in any of (5). (7) The image file generation unit further comprises an image file generation unit that generates an image file containing the image, the 3D information, metadata including confidence information indicating the confidence level of the 3D information, and the signature. (1) The image processing apparatus described in any of (6). (8) The system further comprises a plane determination unit that determines whether the shape of the subject is planar or not based on the 3D information, The signature generation unit generates the signature if the plane determination unit determines that the shape of the subject is not flat. (1) The image processing apparatus described in any of (7). (9) A key generation unit that generates a key corresponding to the image processing device, A providing unit that provides the aforementioned key to the server. An image processing apparatus according to any one of (1) to (8), further comprising: (10) An image of the subject is obtained by capturing an optical image from the subject, 3D information is acquired from the optical image using the same optical axis as the aforementioned image. Generate signatures for the aforementioned image and the aforementioned 3D information. Image processing methods.

[0372] (11) Image verification processing unit confirms the authenticity of the image by comparing the image with 3D information acquired on the same optical axis as the image. An image processing device equipped with the following features. (12) The image verification processing unit compares the surface irregularities of the subject in the image detected from the image with the surface irregularities of the subject based on the 3D information. (11) The image processing apparatus described above. (13) The image verification processing unit determines the authenticity of the image based on the results of the comparison. (12) The image processing apparatus described above. (14) The image verification processing unit performs display control to display the comparison result on the display unit. (12) The image processing apparatus described above. (15) The system further comprises a signature verification unit that verifies the validity of the signatures of the image and the 3D information, The image verification processing unit determines that the image is not authentic if it cannot confirm the validity of the signature. The image processing apparatus described in any of (11) to (14). (16) The 3D information is further provided with an optical axis determination unit that determines whether the 3D information was acquired on the same optical axis as the image, The image verification processing unit determines that the image is not authentic if the 3D information was not acquired on the same optical axis as the image. The image processing apparatus described in any of (11) to (15). (17) The system further comprises a reliability determination unit that determines whether the 3D information is reliable based on reliability information indicating the reliability of the 3D information, which is stored in the image file that stores the image and the 3D information, The image verification processing unit determines that the image is not authentic if the 3D information is unreliable. The image processing apparatus described in any of (11) to (16). (18) The system further comprises a reliability calculation unit that calculates the reliability of the 3D information based on camera parameters relating to the image, which are stored in an image file containing the image and the 3D information. The image processing apparatus described in any of (11) to (17). (19) The system further comprises a shutter speed determination unit that determines whether the shutter speed of the image at the time the image was generated is faster than a predetermined standard, based on camera parameters relating to the image stored in an image file that stores the image and the 3D information. The image verification processing unit determines that the image is not authentic if the shutter speed is slower than the standard. The image processing apparatus described in any of (11) to (18). (20) The authenticity of the image is confirmed by comparing the image with 3D information acquired on the same optical axis as the image. Image processing methods. [Explanation of symbols]

[0373] 100 Image processing system, 110 Network, 111 Imaging device, 112 Server, 113 Terminal device, 121 Control unit, 122 Imaging processing unit, 131 Storage unit, 132 Key generation unit, 133 Upload unit, 134 Recording unit, 141 Optical system, 142 Image sensor, 143 RAW processing unit, 144 YUV processing unit, 145 Reduced image generation unit, 146 Metadata addition unit, 147 Hash processing unit, 148 3D information sensor, 149 3D information processing unit, 150 Signature generation unit, 151 Image file generation unit, 153 Signature control unit, 154 Confidence calculation unit, 161 Sensor unit, 171 Image plane phase difference pixel, 201 Control unit, 221 Communication unit, 222 Image analysis engine, 223 231 Device public key database, 232 Device public key management unit, 233 Signature verification unit, 233 Image verification processing unit, 234 Verification control unit, 235 Optical axis determination unit, 236 Confidence determination unit, 237 Confidence calculation unit, 238 Shutter speed determination unit, 239 Development processing unit, 241 Image verification unit, 242 Support processing unit, 301 CPU, 302 ROM, 303 RAM, 304 Bus, 310 Input / Output interface, 311 Input unit, 312 Output unit, 313 Storage unit, 314 Communication unit, 315 Drive, 321 Removable recording medium

Claims

1. An image acquisition unit that acquires an image of the subject by capturing an optical image from the subject, A 3D information acquisition unit acquires 3D information showing the surface irregularities of the subject from the optical image using the same optical axis as the aforementioned image, A plane determination unit determines whether the shape of the subject is planar or not based on the 3D information, If the plane determination unit determines that the shape of the subject is not flat, the signature generation unit generates a signature for the image and the 3D information. An image processing device equipped with the following features.

2. The 3D information acquisition unit acquires the 3D information using a phase difference method. The image processing apparatus according to claim 1.

3. The image acquisition unit has a pixel array in which pixels that convert light from the subject into photoelectric energy are arranged in a matrix, The 3D information acquisition unit has a plurality of image plane phase difference detection pixels formed in the pixel array, and uses the image plane phase difference detection pixels to detect phase difference data for multiple locations in the image acquired by the pixel array, and uses the detected plurality of phase difference data as the 3D information. The image processing apparatus according to claim 2.

4. The 3D information acquisition unit acquires the 3D information within a predetermined range corresponding to the field of view of the image. The image processing apparatus according to claim 1.

5. The signature generation unit encrypts the hash value of the image and the information including the 3D information to generate the signature. The image processing apparatus according to claim 1.

6. The image acquisition unit acquires a moving image of the subject, The 3D information acquisition unit acquires the 3D information from the optical image on the same optical axis as the moving image. The image processing apparatus according to claim 1.

7. The image acquisition unit acquires the image from one of the optical images that have been split by the beam splitter. The 3D information acquisition unit acquires the 3D information from the other half of the optical image that has been split by the beam splitter. The image processing apparatus according to claim 1.

8. The 3D information acquisition unit acquires the 3D information at the exposure start timing when the main exposure begins during the imaging by the image acquisition unit. The image processing apparatus according to claim 1.

9. The 3D information acquisition unit acquires the 3D information multiple times during a single image capture by the image acquisition unit. The image processing apparatus according to claim 8.

10. The unit further comprises an image file generation unit that generates an image file containing the aforementioned image, the 3D information, and the signature. The image processing apparatus according to claim 1.

11. The system further comprises a reliability calculation unit that calculates the reliability of the 3D information based on the aforementioned image and the aforementioned 3D information. The image processing apparatus according to claim 1.

12. The system further includes a display unit that displays the reliability of the aforementioned image and the aforementioned 3D information. The image processing apparatus according to claim 11.

13. The unit further comprises an image file generation unit that generates an image file containing the aforementioned image, the aforementioned 3D information, metadata including information indicating the reliability of the aforementioned 3D information, and the aforementioned signature. The image processing apparatus according to claim 11.

14. An image processing and editing unit that performs processing and editing on the aforementioned image, A 3D information processing and editing unit performs the same processing and editing on the 3D information as was done on the image. Furthermore, The signature generation unit generates signatures for the processed and edited image and the 3D information. The image processing apparatus according to claim 1.

15. The system further comprises an image file generation unit that generates an image file containing the processed and edited image, the 3D information, the signature, and metadata indicating the content of the processed and edited image. The image processing apparatus according to claim 14.

16. An image of the subject is obtained by capturing an optical image from the subject. 3D information showing the surface irregularities of the subject is obtained from the optical image using the same optical axis as the aforementioned image. Based on the 3D information, it is determined whether the shape of the subject is planar or not. If it is determined that the shape of the subject is not flat, a signature is generated for the image and the 3D information. Image processing methods.

17. An optical axis determination unit that determines whether 3D information showing the surface irregularities of a subject was acquired on the same optical axis as the image of the subject, A signature verification unit that verifies the validity of the signature of the information including the aforementioned image and the aforementioned 3D information, If the validity of the signature is confirmed, the image verification processing unit will verify the authenticity of the image by comparing the surface irregularities of the subject estimated from the image with the surface irregularities of the subject indicated by the 3D information, and will determine that the image is not authentic if the 3D information was not acquired on the same optical axis as the image. An image processing device equipped with the following features.

18. The image verification processing unit performs display control to display the comparison result on the display unit. The image processing apparatus according to claim 17.

19. The system further includes a reliability determination unit that determines whether the 3D information is reliable based on reliability information indicating the reliability of the 3D information, which is stored in the image file containing the image and the 3D information. The image verification processing unit determines that the image is not authentic if the 3D information is unreliable. The image processing apparatus according to claim 17.

20. The system further includes a reliability calculation unit that calculates the reliability of the 3D information based on camera parameters relating to the image, which are stored in an image file containing the image and the 3D information. The image verification processing unit verifies the authenticity of the image based on the reliability of the 3D information. The image processing apparatus according to claim 17.

21. If the signature verification unit fails to confirm the validity of the signature, it updates the metadata of the image and re-verifies the validity of the signature. The image processing apparatus according to claim 17.

22. Determine whether 3D information showing the surface irregularities of the subject was acquired on the same optical axis as the image of the subject, The validity of the signature of the information including the aforementioned image and the aforementioned 3D information is confirmed. If the validity of the signature is confirmed, the authenticity of the image is confirmed by comparing the surface irregularities of the subject estimated from the image with the surface irregularities of the subject indicated by the 3D information. If the 3D information was not acquired on the same optical axis as the image, the authenticity of the image is determined to be lacking. Image processing methods.

Images