Image generation method, image file creating method, and image generation device

Abstract

Provided are an image generation method, an image file, and an image generation device with which it is possible to appropriately generate information pertaining to a basic image used for generating a synthesized image.　An image generation method according to an embodiment of the present invention comprises: an accepting step for accepting any desired image generation mode from a plurality of image generation modes when generating a synthesized image by synthesizing the plurality of basic images; and a generating step for generating an attendant image as attendant information to the synthesized image, in accordance with the image generation mode accepted in the accepting step and on the basis of at least one of the plurality of basic images.

Description

Image generation method, image file creation method, and image generation device

[0001] One embodiment of the present invention relates to an image generation method, an image file creation method, and an image generation device.

[0002] An image file of a captured image (main image) may store additional information about the main image. An additional image, such as a thumbnail image, may be generated as additional information for the main image, and in this case, the generated additional image is stored in the image file (see Patent Document 1). The additional image is used, for example, to easily check the contents of the main image.

[0003] JP 2009-20266 A

[0004] Incidentally, the main image included in an image file may be a composite image formed by combining multiple images. In this case, information indicating the type of image before the combination (basic image) is useful as supplementary information about the composite image, which is the main image, and is preferably stored in the image file. In addition, it is necessary to appropriately generate information about the basic image, taking into account the method by which the composite image was generated, i.e., the type of generation mode for the composite image.

[0005] One embodiment of the present invention solves the problems of the conventional technology described above, and aims to provide an image generation method, an image file, and an image generation device that can appropriately generate information about the basic images used to generate a composite image.

[0006] In order to achieve the above-mentioned object, an image generation method according to one embodiment of the present invention comprises a reception process for accepting any image generation mode from a plurality of image generation modes when generating a composite image by combining a plurality of basic images, and a generation process for generating an ancillary image as ancillary information of the composite image based on at least one of the plurality of basic images in accordance with the image generation mode accepted in the reception process.

[0007] Furthermore, the image generating method according to one embodiment of the present invention may include an image determining step of determining, in accordance with a user's designation, a basic image to be used in generating an incidental image from among a plurality of basic images. In this case, the generating step may generate an incidental image corresponding to the basic image determined in the image determining step.

[0008] The auxiliary image may be an image that does not change in accordance with the change in the composite image due to the correction process.

[0009] The plurality of image generation modes may include a first mode in which a composite image is generated by combining a plurality of basic images that are captured under different imaging conditions of the imaging device at the time of image capture. If the first mode is accepted in the accepting step, one or more accessory images may be generated in the generating step based on some or all of the plurality of basic images.

[0010] In the above configuration, the imaging condition may be an exposure amount. When the first mode is accepted in the accepting step, the generating step may generate two or more auxiliary images based on at least two or more of a plurality of basic images having different exposure amounts at the time of shooting.

[0011] Furthermore, when the first mode is accepted in the accepting step, in the generating step, an auxiliary image based on each of the plurality of basic images may be generated for each basic image.

[0012] The plurality of image generation modes may also include a second mode in which a composite image is generated by combining a plurality of basic images captured by shifting the imaging position of the imaging element within the imaging device. If the second mode is accepted in the accepting step, an incidental image may be generated in the generating step based on some of the plurality of basic images.

[0013] The plurality of image generation modes may also include a third mode in which a composite image is generated by combining a plurality of basic images captured by an imaging device at different locations. If the third mode is received in the receiving step, an incidental image may be generated in the generating step based on all of the plurality of basic images.

[0014] The plurality of image generation modes may also include a fourth mode in which a composite image is generated by arranging and compositing a plurality of basic images. If the fourth mode is accepted in the accepting step, the generation step may generate an auxiliary image based on a portion of the plurality of basic images.

[0015] In addition, in the generating step, some basic images may be selected from the plurality of basic images based on the sizes of the plurality of basic images in the composite image, and the accessory image may be generated based on the selected some basic images.

[0016] In addition, in the generating step, an auxiliary image based on at least one of the plurality of basic images and an auxiliary image based on the composite image may be generated.

[0017] Furthermore, information indicating whether a plurality of basic images are images to be used for generating an incidental image may be stored for each of the plurality of basic images. In this case, in the generating step, the incidental image may be generated based on a basic image selected from the plurality of basic images based on the information.

[0018] Furthermore, an image file creation method according to one embodiment of the present invention is an image file creation method that creates an image file that includes a composite image as a main image and an auxiliary image generated by one of the image generation methods described above.

[0019] Furthermore, an image generating device according to one embodiment of the present invention is an image generating device that includes a processor, and when generating a composite image by combining a plurality of basic images, the processor accepts any image generation mode from a plurality of image generation modes, and generates an ancillary image as ancillary information of the composite image based on at least one of the plurality of basic images in accordance with the accepted image generation mode.

[0020] 1 is a diagram showing an example of image data structure; an explanatory diagram of an image file; an explanatory diagram of development correction processing; an explanatory diagram of color correction processing; an explanatory diagram of processing correction processing; a diagram showing an example of the configuration of an image file creation device according to an embodiment of the present invention; a diagram showing an example of a screen for accepting a type of collateral image; a diagram showing functions of an image file creation device according to the first embodiment; a diagram showing an example of a screen for selecting whether or not to acquire a third collateral image; a diagram showing a first example of a generation pattern of an collateral image; a diagram showing a second example of a generation pattern of an collateral image; a diagram showing a third example of a generation pattern of an collateral image; an explanatory diagram of a first thumbnail image; a diagram showing an image file creation flow according to the first embodiment (part 1); a diagram showing an image file creation flow according to the first embodiment (part 2); an explanatory diagram of a first mode compositing process; an explanatory diagram of a second mode compositing process; an explanatory diagram of a third mode compositing process; an explanatory diagram of a fourth mode compositing process; a diagram showing functions of an image file creation device according to a second embodiment as an embodiment of the present invention; a diagram showing an example of a screen for accepting designation of an image generation mode; a diagram showing an example of a screen for accepting designation of a basic image used to generate an collateral image. FIG. 10 is an explanatory diagram of a thumbnail image generated when the image generation mode is the first mode. FIG. 11 is an explanatory diagram of a thumbnail image generated when the image generation mode is the second mode. FIG. 12 is an explanatory diagram of a thumbnail image generated when the image generation mode is the third mode. FIG. 13 is an explanatory diagram of a thumbnail image generated when the image generation mode is the fourth mode. FIG. 14 is a diagram showing an image file creation flow according to a second embodiment as an embodiment of the present invention. FIG. 15 is an explanatory diagram of the effectiveness of an image file creation method according to a third embodiment. FIG. 16 is a diagram showing collateral images generated in the third embodiment. FIG. 17 is a diagram showing functions of an image file creation device according to the third embodiment. FIG. 18 is a diagram showing an example of a screen for specifying the type of main subject. FIG. 19 is a diagram showing an example of a screen for accepting selection of a mode for increasing the resolution or number of gradations of collateral images. FIG. 19 is a diagram showing an image file creation flow according to the third embodiment. FIG. 19 is a diagram showing functions of an image file creation device according to a fourth embodiment.FIG. 10 is a diagram showing an incidental image generated in the fourth embodiment; FIG. 11 is a diagram showing an example of a screen for accepting a setting for increasing the resolution of an incidental image; FIG. 12 is a diagram showing a procedure for generating an incidental image in the fourth embodiment; FIG. 13 is a diagram showing the structure of an image file created in the fourth embodiment; and FIG. 14 is a diagram showing an image file creation flow according to the fourth embodiment.

[0021] Specific embodiments of the present invention will be described. However, the embodiments described below are merely examples given to facilitate understanding of the present invention and are not intended to limit the present invention. In other words, the present invention can be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0022] In addition, in this specification, the concept of "device" includes not only a single device that performs a specific function, but also a combination of multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function.

[0023] In this specification, the term "user" refers to a person who uses the image file creation device of the present invention, specifically, for example, a person who uses an imaging device (described below) equipped with the image file creation device of the present invention. In this specification, the term "person" refers to an entity that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, as well as computers and devices that constitute artificial intelligence (AI). AI is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. The algorithm of AI is arbitrary, and may be, for example, an expert system, case-based reasoning (CBR), a Bayesian network, or a subsumption architecture.

[0024] Also, in this specification, "image" means data (image data) that indicates the gradation of each pixel, and the main example is still image data, but it may also be video data made up of multiple frame images.

[0025] <<Outline of Specific Embodiments of the Present Invention>> One embodiment of the present invention relates to an image generation method, an image file creation method, and an image generation device.

[0026] Image files are created by known imaging devices such as digital cameras. Alternatively, an information processing terminal may create image files based on images captured by the imaging device or received from a device on a network. Image files are created in accordance with a file format adopted by the device creating the image files. Examples of file formats include JPEG (Joint Photographic Experts Group), Tiff (Tagged Image File Format), GIF (Graphics Interchange Format), BMP (Microsoft Windows Bitmap Image), PNG (Portable Network Graphics), and HEIF (High Efficiency Image File Format). The file format is reflected in the data structure of the image file. For example, in JPEG, an image file begins with a Start of Image (SOI) marker segment and ends with an End of Image (EOI) marker segment, as shown in FIG. 1 .

[0027] As shown in FIG. 2 , an image file includes a main image MP and additional information AD. The main image MP is the primary image in the image file, and is, for example, the image with the highest resolution or data capacity in the image file. The main image MP is acquired, for example, by capturing an image of a subject using an imaging device. Specifically, when a user presses the release button of the imaging device, the imaging element of the imaging device generates and outputs an analog image signal using an optical signal received from the subject. The analog image signal is converted into a RAW image, which is a digital image. Correction processing such as gamma correction is performed on the RAW image, and the corrected RAW image is compressed using a predetermined compression method. This develops the RAW image, and compressed image data representing the developed image is created. Note that the number of gradations (bits) of the RAW image may be greater than the number of gradations (bits) of the developed image.

[0028] The subject in the main image MP is an object to be captured that exists within the imaging range of the imaging device. It is not limited to specific tangible objects, but also includes intangible objects such as landscapes, scenes, or patterns. The main image MP also changes when a correction process is performed. Image correction is data processing that modifies the image's content, color, or image quality. Image content refers to the subject and background of the image, as well as the image's angle of view. The angle of view is the data processing range within which the image is displayed or rendered, and this range is defined as a two-dimensional coordinate space with two orthogonal axes as coordinate axes. The color of an image is represented by the gradations (color gradations) of the multiple pixels that make up the image. The image quality is determined by various factors, including resolution, number of gradations, color gamut, spatial frequency, and dynamic range.

[0029] The correction process may be performed within the imaging device that captured the main image MP, or may be performed by an information processing terminal or the like that imports the main image MP from the imaging device. The correction process may also be performed based on a user's operation, or may be automatically performed by a function of the imaging device or the information processing terminal. The main image MP that is the target of the correction process may include a raw image before development.

[0030] In one embodiment of the present invention, a development correction process, a color correction process, or a processing correction process can be performed on the main image MP. As shown in FIG. 3A , the development correction process is a development correction process performed on a RAW image serving as the main image. Specifically, correction processes such as gamma correction for developing a RAW image so that it can be displayed correspond to the development correction process. Hereinafter, an image obtained by performing the development correction process on a RAW image serving as the main image will be referred to as a "developed image."

[0031] The color correction process is a correction process that includes multiple color corrections and is performed on the main image MP, for example, on the developed image. In the main image MP that has undergone the color correction process, the color of the corrected areas changes from the pre-correction stage, as shown in FIG. 3B . Here, the development correction process described above does not include all of the multiple color corrections that can be performed in the color correction process. In other words, the development correction process is a correction process that is simpler than the color correction process.

[0032] Examples of color correction include white balance correction, color gradation correction, and color reproduction correction. Color reproduction correction adjusts the hue, gradation, saturation, contrast, etc. of an image based on a reproduction style selected by the user from among multiple reproduction styles related to the color of the image. Color reproduction styles are classified based on the overall impression or atmosphere of the image, and multiple color reproduction styles are prepared in advance on the device, allowing the user to select any one. The type of correction to be performed in each color correction may be determined according to the imaging conditions, for example, based on user operation. For example, if the user selects white balance correction, white balance correction may be performed according to the light source used at the time of imaging. Alternatively, the correction content in the color correction may be determined solely based on user operation. Hereinafter, an image obtained by performing color correction processing on a main image will be referred to as a "color-corrected image."

[0033] The processing and correction process is a correction process for processing a subject in a main image, and is performed on the main image MP, for example, on a developed image or a color-corrected image. Processing and correction processes include trimming processes for extracting an area including the subject from the main image (hereinafter referred to as the subject area), and processes for adding text or illustrations to the main image to be corrected. Processing and correction processes may also include processes for combining part or all of another image with the main image, as shown in FIG. 3C. Hereinafter, an image obtained by performing processing and correction processes on a main image will be referred to as an "processed image."

[0034] In one embodiment of the present invention, one image file contains only one main image MP, and the main image MP corresponds to any of the developed image, color-corrected image, or processed image described above. Therefore, when a color correction process is performed on the developed image serving as the main image MP contained in the image file, the main image MP in the image file is replaced (overwritten) from the developed image with the color-corrected image. Furthermore, when the main image MP contained in the image file is a developed image or color-corrected image and a processing correction process is performed on the main image MP, the main image MP in the image file is replaced from the developed image or color-corrected image with the processed image. Note that if the main image MP changes due to the correction process, the main image MP before correction (before replacement) may be deleted from the image file. Alternatively, the main image MP before correction may remain in the image file as a backup image (thumbnail image).

[0035] The additional information AD is information related to the main image MP. More specifically, the additional information AD can be written in the header area of ​​an image file. For example, in JPEG XT Part 3, a type of JPEG, as shown in FIG. 1, marker segments "APP1" and "APP11" are provided as header areas where additional information can be written. "APP1" stores additional information related to the capture date and time, capture location, capture conditions, etc. of the main image MP as tag information. "APP11" includes JUMBF (JPEG Universal Metadata box format) boxes, specifically JUMBF1 and JUMBF2 boxes, which are metadata storage areas. An example of a JPEG standard for additional information is Exif (Exchangeable image file format).

[0036] As shown in Fig. 1, thumbnail images are stored as incidental information in APP11. Thumbnail images correspond to incidental images, and are created by thinning out the main image MP, with a resolution lower than that of the main image MP. In other words, by lowering the resolution below that of the main image MP, thumbnail images TP are generated that are reduced versions of the main image MP, as shown in Fig. 2, and these thumbnail images TP are stored in APP11 as incidental information.

[0037] In one embodiment of the present invention, a thumbnail image TP is acquired in response to changes in the main image MP due to the correction process. Specifically, a thumbnail image TP based on the changed main image MP is generated in conjunction with the changes in the main image MP. Specifically, a thumbnail image TP based on a developed image (hereinafter referred to as a developed thumbnail image) is generated in response to changes in the main image MP (specifically, a RAW image) due to the development correction process. Furthermore, a thumbnail image TP based on a color-corrected image (hereinafter referred to as a color-corrected thumbnail image) is generated in response to changes in the main image MP due to the color correction process. Furthermore, a thumbnail image TP based on a processed image (hereinafter referred to as a processed thumbnail image) is generated in response to changes in the main image MP due to the processing correction process.

[0038] The developed thumbnail image, color-corrected thumbnail image, and processed thumbnail image represent types of accessory images. In one embodiment of the present invention, the user can specify which type of thumbnail image (accessory image) to acquire. Then, the thumbnail image of the type specified by the user is acquired, and the acquired thumbnail image is stored in the image file as accessory information for the main image MP.

[0039] In one embodiment of the present invention, generating a thumbnail image corresponds to acquiring a thumbnail image. However, the present invention is not limited to this, and a thumbnail image may be acquired by first generating a plurality of thumbnail images and then selecting one of the thumbnail images to store in an image file.

[0040] The thumbnail image TP may be a reduced image of the entire main image MP. Alternatively, the thumbnail image TP may be a reduced image of a portion of the main image MP, for example, an extracted image obtained by extracting the area of ​​the main subject. The main subject is a subject selected from among the subjects in the main image MP based on, for example, the focus position of the imaging device at the time of image capture or its positional relationship with the user's line of sight. The main subject may also be selected based on the image quality, such as the resolution of the main image MP (specifically, the degree of blur or blur, etc.) or clarity. Alternatively, a subject specified by the user in the main image MP may be selected as the main subject.

[0041] The compression method of the thumbnail image TP may be the same as or different from the compression method of the main image MP from which the thumbnail image TP is derived. The image from which the thumbnail image TP is derived (hereinafter also referred to as the original image) is the main image MP at the time the thumbnail image TP is generated.

[0042] Furthermore, the number of thumbnail images TP that can be stored in one image file only needs to be one or more, and multiple thumbnail images TP may be stored in one image file. In other words, if two or more types of thumbnail images are generated from among a developed thumbnail image, a color-corrected thumbnail image, and an edited thumbnail image, the image file may contain these two or more types of thumbnail images. Alternatively, the number of thumbnail images TP that can be stored in an image file may be only one.

[0043] The additional information AD may store management information related to the thumbnail images TP, as shown in Fig. 2. The management information includes, for example, information related to the creation history (creation date and time) of the thumbnail images, and information related to the type of thumbnail image. The information related to the type of thumbnail image indicates whether the thumbnail image stored in the image file is a developed thumbnail image, a color-corrected thumbnail image, or an edited thumbnail image. This information may be text information describing the type of thumbnail image, or information consisting of an identification code corresponding to the type of thumbnail image.

[0044] The incidental information AD may also include information obtained by encrypting the thumbnail image in order to prevent tampering with the thumbnail image, as shown in FIG. 2. Specifically, the incidental information AD may include a hash value obtained by hashing the thumbnail image. The information stored in the image file in order to prevent tampering is not limited to the hash value, and may be information obtained by encrypting or encoding the thumbnail image using a method other than hashing. The incidental information AD may also include information regarding the type of main image MP stored in the image file, as shown in FIG. 2. This information indicates whether the main image MP is a developed image, a color-corrected image, or an edited image.

[0045] <<Example of the configuration of an image file creation device according to one embodiment of the present invention>> An image file creation device according to one embodiment of the present invention (hereinafter referred to as image file creation device 10) comprises a processor 11, a memory 12, and a communication interface 13, as shown in Figure 4.

[0046] The processor 11 is configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a tensor processing unit (TPU). The memory 12 is configured, for example, by semiconductor memory such as a read-only memory (ROM) and a random access memory (RAM). The communication interface 13 is configured, for example, by a network interface card or a communication interface board.

[0047] A program for creating an image file (hereinafter referred to as an image file creation program) is stored in the memory 12. The image file creation program is a program for causing the processor 11 to execute each step of an image file creation method described below. The image file creation program may be obtained by reading it from a computer-readable recording medium, or by downloading it via a communication network such as the Internet or an intranet.

[0048] The image file creation device 10 can communicate with other devices through a communication interface 13 and send and receive data to and from those devices. The image file creation device 10 also includes an input device 14 and an output device 15, as shown in Fig. 4. The input device 14 includes devices that accept user operations, such as a touch panel and cursor buttons. The output device 15 includes a display device, such as a monitor.

[0049] The image file creation device 10 can freely access various data stored in the storage 16. The data stored in the storage 16 includes data necessary for creating image files, such as images, image capture conditions, information about image correction processing, and information about operations received from the user. The storage 16 also stores created image files. The storage 16 may be built into or external to the image file creation device 10, or may be configured as a network-attached storage (NAS) or the like. Alternatively, the storage 16 may be an external device, such as an online storage device, that can communicate with the image file creation device 10 via the Internet or a mobile communication network.

[0050] In one embodiment of the present invention, the image file creation device 10 is mounted on an imaging device such as a digital camera, as shown in Fig. 4. The mechanical configuration of an imaging device (hereinafter referred to as imaging device 20) equipped with the image file creation device 10 is substantially the same as that of known imaging devices that have the function of creating image files.

[0051] That is, the imaging device 20 captures an image of a subject according to preset imaging conditions to obtain a RAW image as the main image MP. The imaging conditions include exposure time (exposure amount), ISO sensitivity, focus position at the time of capture, focal length, etc. The imaging device 20 then develops the RAW image and performs color correction or processing correction on the developed image as appropriate. This results in the main image MP. The imaging device 20 also generates additional information such as a thumbnail image TP based on the main image MP and creates an image file including the main image MP and the additional information. In other words, the imaging device 20 constituting the image file creation device 10 functions as an image generation device that generates and obtains the main image MP and thumbnail image TP.

[0052] In one embodiment of the present invention, the imaging device 20 may have an autofocus (AF) function that automatically focuses on a predetermined position within the imaging range during imaging, and a function that identifies a focal position (AF point). The AF point is identified as a coordinate position with a reference position within the imaging range as the origin. The imaging device 20 may also have a viewfinder through which the user looks during imaging. In this case, the imaging device 20 may have a function that detects the respective positions of the user's line of sight and pupil while using the viewfinder and identifies the user's line of sight. The user's line of sight corresponds to the intersection position between the user's line of sight as they look into the viewfinder and the display screen within the viewfinder.

[0053] In one embodiment of the present invention, the imaging device 20 may have a so-called pixel shift function, which is a function for changing the position of the imaging element (i.e., the imaging position) by shifting the imaging element by one pixel or half a pixel in the vertical and horizontal directions within the imaging device 20. Here, the imaging position refers to the position of each pixel in the imaging element.

[0054] In one embodiment of the present invention, the imaging device 20 may have a function for capturing images in live view mode before capturing the main image MP. With this function, for example, when a user half-presses the release button as an operation for preparing for imaging, this triggers continuous image capture of the subject within the imaging range at a predetermined time interval. Images captured in live view mode (live view images) are displayed in real time as through images on the display of the imaging device 20. The interval between images in live view mode may be constant, or the interval may vary depending on the imaging conditions, etc.

[0055] Furthermore, the live view image is an image captured in the same imaging range as the main image before capturing the main image, and is temporarily stored in the memory 12 or the storage 16. The live view image temporarily stored in the memory 12 or the storage 16 may be all or some of the live view images captured before capturing the main image.

[0056] In one embodiment of the present invention, the imaging device 20 may have a function for continuing imaging in the quick view mode after capturing the main image. This function allows the subject in the imaging device to be captured a certain amount of time after the user presses the release button. Images captured in the quick view mode (quick view images) are temporarily stored in the memory 12 or the storage 16. Note that imaging in the quick view mode may continue to capture images of the subject within the imaging range at predetermined time intervals after capturing the main image. In this case, the interval between images in the quick view mode may be constant or may vary depending on the imaging conditions, etc.

[0057] In one embodiment of the present invention, the imaging device 20 may have a function of analyzing the main image and detecting a subject area in the main image. A known subject detection technique may be used as a method for detecting the subject area. In one embodiment of the present invention, the imaging device 20 may have a function of analyzing the detected subject area and identifying the resolution (specifically, the degree of blur or shake, etc.) or sharpness of the subject.

[0058] Furthermore, in one embodiment of the present invention, the image capture device 20 may have a function to identify the type of subject present in the subject area, and may also have a function to identify the facial expression if the subject is a person. As a method for identifying the type or facial expression of the subject, for example, known subject recognition technology can be used.

[0059] Furthermore, in one embodiment of the present invention, the imaging device 20 may have a function of calculating a score for the subject region (i.e., the image of the subject) based on the resolution, clarity, facial expression, etc. of the identified subject. As a method of calculating a score from each evaluation item related to the image of the subject, for example, the technology described in JP 2019-118021 A can be used.

[0060] <<Generation of Thumbnail Image>> In one embodiment of the present invention, a thumbnail image TP is acquired (generated) when a correction process is performed on the main image MP within the imaging device 20 that captured the main image MP. For example, when a development correction process is performed on a RAW image serving as the main image MP, a developed thumbnail image is generated based on the developed main image, i.e., the developed image.

[0061] Furthermore, in one embodiment of the present invention, it is possible to set which correction process is performed on the main image MP to generate a thumbnail image, in other words, the type of thumbnail image to be generated. Specifically, before the main image MP is captured, the setting screen shown in FIG. 5 is displayed on the display of the imaging device 20. This setting screen displays selectable types of thumbnail images: developed thumbnail image, color-corrected thumbnail image, and processed thumbnail image. The user specifies one or more types of thumbnail image on the setting screen.

[0062] Then, when the user specifies a type of thumbnail image and a correction process corresponding to the specified type is performed on the main image MP, a thumbnail image of that type is acquired. For example, if the user specifies a "color-corrected thumbnail image" and then a color correction process is performed on the main image MP, a color-corrected thumbnail image is generated and acquired.

[0063] Although the setting screen in FIG. 5 displays selectable types of thumbnail images, the display content is not particularly limited as long as the type of thumbnail image to be generated can be identified. For example, explanatory text for each correction process or sample images after each correction process may be selectably displayed on the screen. In this case, when the user selects one of the correction processes and the selected type of correction process is performed on the main image MP, a thumbnail image corresponding to that correction process is generated. By receiving the type of correction process from the user in this way, the type of thumbnail image to be generated may be indirectly received.

[0064] Furthermore, the type of thumbnail image does not necessarily have to be accepted in advance before the main image MP is captured, but may be accepted, for example, at the time of or immediately after the execution of a correction process on the main image MP. Furthermore, each time a correction process is executed on the main image MP, the user may be asked whether or not to generate a thumbnail image corresponding to the correction process.

[0065] In one embodiment of the present invention, the thumbnail images include a first thumbnail image. The first thumbnail image is generated by reducing all or part of the main image at the time the thumbnail image is generated. Furthermore, when a correction process is performed on the main image, which is its original image, the first thumbnail image is an image that does not change in response to changes in the main image caused by the correction process. For example, suppose a developed thumbnail image is generated as the first thumbnail image. In this case, even if a color correction process or a processing correction process is performed on the developed image, which is its original image, after the developed thumbnail image is generated, the developed thumbnail image is not corrected and remains in the state it was in at the time of creation.

[0066] By generating (obtaining) the first thumbnail image, the first thumbnail image can be used as evidence proving the existence of the main image, which is the original image. In other words, even if the main image MP, which is the original image, is changed by a correction process after the first thumbnail image is generated, the original main image MP before correction can be confirmed by referring to the first thumbnail image. As a result, if an unintentional correction process, such as tampering, is performed on the main image MP, the user can notice the tampering by looking at the first thumbnail image generated based on the main image MP before correction.

[0067] The configuration in which the first thumbnail image is not corrected in response to the correction of the main image MP is not particularly limited. For example, the first thumbnail image may be stored in an area in the image file that cannot be corrected (changed), specifically in the header area "APP11" described above, thereby preventing the first thumbnail image from being corrected. Furthermore, information that restricts the correction of the first thumbnail image may be stored in the header area described above, thereby preventing the first thumbnail image from being corrected. Here, the information that restricts the correction of the first thumbnail image may be information obtained by encrypting the first thumbnail image (e.g., a hash value, etc.).

[0068] In one embodiment of the present invention, the thumbnail images further include a second thumbnail image in addition to the first thumbnail image described above. The second thumbnail image is generated by reducing all or part of the main image MP at the time the thumbnail image is generated. Furthermore, when a correction process is performed on the main image MP, which is its original image, the second thumbnail image is an image that changes in response to changes in the main image MP due to the correction process. In other words, the second thumbnail image corresponds to a variable collateral image that changes in response to changes in the main image MP due to the correction process.

[0069] <<Image File Creation Devices According to Each Embodiment Related to the Present Invention>> The functions and operations of image file creation devices according to embodiments related to the present invention will be described using several specific examples. For the second and subsequent embodiments, differences from the previous embodiments will be mainly described, and descriptions of common features will be omitted as appropriate. Furthermore, an embodiment of the present invention may be a combination of two or more of the multiple embodiments (first to fourth embodiments) described below.

[0070] [Regarding the First Embodiment] As shown in FIG. 6 , the image file creation device 10 according to the first embodiment of the present invention includes a type acceptance unit 21, a first generation unit 22, a correction unit 23, a determination unit 24, a selection acquisition unit 25, a second generation unit 26, and a creation unit 27. These functional units are realized by cooperation between the hardware devices of the image file creation device 10 described above and software including the image file creation program described above. In addition, some functions may be realized using artificial intelligence (AI). Each functional unit will be described below.

[0071] (Type Receiving Unit) The type receiving unit 21 receives the type of thumbnail image TP in association with the correction process for the main image MP. For example, the type receiving unit 21 displays the setting screen of FIG. 5 on the display of the imaging device 20. The screen displays developed thumbnail image, color-corrected thumbnail image, and processed thumbnail image as selectable types of thumbnail image. When the user specifies at least one of these, the type receiving unit 21 receives the result of the specification as the type of thumbnail image. At this time, the user can specify two or more types from the three types of thumbnail image, in which case the type receiving unit 21 receives two or more types.

[0072] The timing at which the type accepting unit 21 accepts the type of thumbnail image can be determined arbitrarily as long as it is before a thumbnail image corresponding to that type is generated. Furthermore, when a user specifies a type of correction process for the main image MP, the type accepting unit 21 may accept the specified type of correction process as the type of thumbnail image.

[0073] (First Generation Unit) The first generation unit 22 is triggered by the user fully pressing the release button of the imaging device 20, and controls each unit of the device to capture an image of a subject present within the imaging range of the imaging device 20 and generate a main image MP. At this time, the first generation unit 22 captures an image of the subject according to the set imaging conditions. As a result, a main image MP, more specifically a RAW image, according to the imaging conditions is generated.

[0074] (Correction Unit) The correction unit 23 performs correction processing on the main image MP automatically or in response to a user instruction. For example, when a RAW image is generated as the main image MP, the correction unit 23 performs development correction on the RAW image. Furthermore, when a user instructs color correction on the developed image as the main image MP, the correction unit 23 performs color correction processing on the developed image in accordance with the user instruction. Furthermore, when a user performs processing operations on the developed image or color-corrected image as the main image MP, the correction unit 23 performs processing correction processing on the developed image or color-corrected image based on the user's processing operations.

[0075] In the first embodiment, the image file creation device 10 including the correction unit 23 is mounted on the imaging device 20. Therefore, in the first embodiment, the correction process by the correction unit 23 is performed within the imaging device 20 that captured the main image MP. Because the correction process is performed within the imaging device 20 in this manner, the user can more easily correct the main image MP.

[0076] (Decision Unit) When the correction unit 23 performs a processing correction process on the main image MP, the decision unit 24 decides whether to generate (acquire) a processed thumbnail image, i.e., whether to perform the acquisition process described below. In other words, in the first embodiment, when the processing correction process is performed, a processed thumbnail image is generated only if the decision unit 24 decides to generate a processed thumbnail image. This reduces the burden of data processing compared to generating a processed thumbnail image each time the processing correction process is performed. Furthermore, by limiting the number of processed thumbnail images generated, the required data volume can be reduced.

[0077] In a first example of the first embodiment, the determination unit 24 automatically determines whether to generate an edited thumbnail image (i.e., whether to perform the acquisition process) based on the correction content of the edit correction process. The correction content refers to the content of changes to the subject or angle of view of the main image MP caused by the edit correction process, specifically, the degree of change (correction amount) in the main image MP due to the correction process. The correction content can be quantitatively determined, for example, by comparing the main image MP before and after correction and calculating the amount of change. If the correction content satisfies a predetermined condition, for example, if the correction amount is equal to or greater than a reference value, the determination unit 24 automatically determines to acquire an edited thumbnail image in response to the execution of the edit correction process. Furthermore, in a second example of the first embodiment, when the edit correction process is performed on the main image MP, the determination unit 24 acquires a user's selection and determines whether to generate (acquire) an edited thumbnail image.

[0078] (Selection Acquisition Unit) When the processing and correction process is performed on the main image MP, the selection and acquisition unit 25 automatically determines whether to generate (acquire) an edited thumbnail image via the determination unit 24 or acquires a user selection. In the latter case, when the processing and correction process is performed on the main image MP, a selection screen shown in FIG. 7 is displayed on the display of the imaging device 20. The selection screen displays options for whether to generate an edited thumbnail image. The selection and acquisition unit 25 acquires the user's selection regarding the generation of an edited thumbnail image via the selection screen. Then, an edited thumbnail image is generated only when the processing and correction process is performed and the user selects the generation of an edited thumbnail image. This reduces the data processing burden compared to generating an edited thumbnail image each time the processing and correction process is performed. Furthermore, by reducing the number of edited thumbnail images generated, the required data volume can be reduced.

[0079] As described above, in the first embodiment, whether or not to generate an edited thumbnail image can be determined by the determination unit 24 based on the correction content of the edit correction process. Moreover, whether or not to generate an edited thumbnail image is determined based on the user's selection acquired by the selection acquisition unit 25. Furthermore, when determining whether or not to generate an edited thumbnail image, the user may be able to specify whether to use automatic determination based on the correction content or user selection.

[0080] (Second Generation Unit) The second generation unit 26 generates incidental information related to the main image MP. The incidental information generated by the second generation unit 26 includes a thumbnail image TP. Specifically, when the correction unit 23 executes a correction process, the second generation unit 26 generates a thumbnail image TP based on the corrected main image MP in conjunction with this. As a result, the thumbnail image TP is obtained as incidental information related to the main image MP.

[0081] The thumbnail image TP is generated by reducing all or part of the main image MP at the time of generation. When reducing a part of the main image MP to generate the thumbnail image TP, the second generation unit 26 reduces an area of ​​the main image MP designated by the user or an area selected by the imaging device 20 to generate the thumbnail image TP. Alternatively, the area selected by the imaging device 20 may be, for example, an area in which the main subject of the main image MP is present.

[0082] Furthermore, in the first embodiment, the second generation unit 26 generates the thumbnail image TP based on the type of thumbnail image accepted by the type acceptance unit 21. For example, assume that the type of thumbnail image accepted by the type acceptance unit 21 is a "developed thumbnail image." In this case, as shown in FIG. 8 , when a development correction process is executed on a RAW image serving as the main image MP, the second generation unit 26 generates a developed thumbnail image based on the developed image that is the main image MP after correction.

[0083] As a second example, assume that the types of thumbnail images accepted by the type accepting unit 21 are "developed thumbnail image" and "color-corrected thumbnail image." In this case, as shown in Fig. 9, when a development correction process is performed on a RAW image serving as the main image MP, the second generating unit 26 generates a developed thumbnail image based on the developed image in response to this. Furthermore, when a color correction process is performed on the developed image, the second generating unit 26 generates a color-corrected thumbnail image based on the color-corrected image serving as the main image MP after correction in response to this.

[0084] As a third example, assume that the types of thumbnail images accepted by the type accepting unit 21 are "color-corrected thumbnail image" and "processed thumbnail image." In this case, no thumbnail image (processed thumbnail image) is generated when the development correction process is performed on the RAW image. On the other hand, as shown in FIG. 10 , when the color correction process is performed on the developed image, which is the main image MP, the second generating unit 26 generates a color-corrected thumbnail image based on the color-corrected image in response to this. Furthermore, when the processing correction process is performed on the developed image or the color-corrected image, the second generating unit 26 generates a processed thumbnail image based on the processed image in response to this.

[0085] As described above, in the first embodiment, the thumbnail image TP is generated based on the type of thumbnail image received from the user. As a result, when the correction process corresponding to the type received from the user is performed on the main image MP, a thumbnail image can be generated based on the corrected main image MP and stored in the image file.

[0086] Also, in the first embodiment, as described above, when the processing and correction process is performed on the main image MP, the determination unit 24 determines whether or not to acquire a processed thumbnail image. Then, when the determination unit 24 determines to acquire a processed thumbnail image, the second generation unit 26 generates a processed thumbnail image. Also, in the first embodiment, as described above, when the processing and correction process is performed on the main image MP, the selection acquisition unit 25 acquires a user's selection as to whether or not to acquire a processed thumbnail image. Then, when the selection acquisition unit 25 acquires a selection to acquire a processed thumbnail image, the second generation unit 26 generates a processed thumbnail image in accordance with the selection.

[0087] Furthermore, in the first embodiment, a first thumbnail image is generated (acquired) as the thumbnail image. When a correction process is performed on the main image MP, which is its original image, the first thumbnail image does not change in response to changes in the main image due to the correction process, as shown in FIG. 11 , and remains the image at the time of acquisition. In other words, the first thumbnail image is an image with high reliability (credibility) as evidence proving what the main image MP looked like at the time of its generation. Therefore, even if unintentional corrections, such as tampering, are made to the main image MP, it is possible to recognize that the corrections were made by acquiring a first thumbnail image based on the main image MP before the corrections.

[0088] Furthermore, in the first embodiment, the second generation unit 26 may generate a correctable second thumbnail image together with the first thumbnail image. When a correction process is performed on the main image MP, which is the original image of the second thumbnail image, the second thumbnail image changes in response to changes in the main image MP due to the correction process. By acquiring such a second thumbnail image together with the first thumbnail image, it is possible to compare the two thumbnail images and easily determine whether or not a correction process has been performed on the main image MP, which is the original image. Note that the present invention is not limited to the above case, and the second generation unit 26 may generate only the first thumbnail image without generating a second thumbnail image.

[0089] Furthermore, in the first embodiment, the second generation unit 26 generates management information indicating the type of thumbnail image generated as incidental information related to the main image MP. This makes it easy to recognize what type of thumbnail image has been acquired for the main image MP. Furthermore, in the first embodiment, the second generation unit 26 generates encrypted information about the generated thumbnail image as incidental information related to the main image MP. This information includes a hash value obtained by hashing the thumbnail image. By acquiring encrypted information about the thumbnail image in this manner, it is possible to discover alterations and modifications to the thumbnail image itself. As a result, the authenticity (security) of thumbnail images, particularly the first thumbnail image, can be further improved.

[0090] (Creation Unit) The creation unit 27 creates an image file, specifically, in a file format selected by default or the user. The image file is created, for example, when the main image MP is acquired. Furthermore, each time a correction process is performed on the main image MP, the main image MP in the image file is replaced with the corrected image. For example, suppose that a correction process is performed on a developed image or a color-corrected image serving as the main image MP included in the image file to acquire a processed image. In this case, the creation unit 27 replaces the main image MP included in the image file from the developed image or color-corrected image with the processed image. Note that the developed image or color-corrected image that is the main image MP before replacement may be deleted or may remain in the image file.

[0091] Furthermore, when the second generation unit 26 generates incidental information related to the main image MP, the creation unit 27 includes the generated incidental information in the image file. Specifically, when the second generation unit 26 generates a thumbnail image TP based on the main image MP, the creation unit 27 stores the thumbnail image TP in "APP11", which is the header area of ​​the image file.

[0092] Furthermore, if the correction process is performed multiple times on the main image MP, and each time a correction process is performed, a new thumbnail image TP is created based on the corrected main image, the newly created thumbnail image TP is added to the image file. In other words, in the first embodiment, if two or more types of thumbnail images are generated from among a developed thumbnail image, a color-corrected thumbnail image, and an edited thumbnail image, the image file will contain two or more types of thumbnail images.

[0093] The thumbnail images TP included in the image file include a first thumbnail image that does not change in response to correction of the main image MP. Furthermore, the thumbnail images TP in the image file may also include, in addition to the first thumbnail image, a second thumbnail image as a variable collateral image that changes in response to correction of the main image MP. Furthermore, the image file may also include information other than the thumbnail images TP generated by the second generation unit 26, specifically, information about the type of thumbnail image, hash values ​​of the thumbnail images, etc. (see FIG. 2 ).

[0094] (Example of Operation of Image File Creation Device in First Embodiment) Next, as an example of operation of the image file creation device 10 in the first embodiment, an image file creation flow using this device will be described. The image file creation flow described below uses the image file creation method of the present invention. In other words, each step in the image file creation flow described below corresponds to a component of the image file creation method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order of execution of two steps in the flow may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0095] 12A and 12B by the processor 11 included in the image file creation device 10. That is, in each step of the image file creation flow, the processor 11 executes a process corresponding to each step among the data processes defined in the image file creation program.

[0096] In the image file creation flow according to the first embodiment, the processor 11 first performs a type reception process (S001). In the type reception process, the screen shown in FIG. 5 is displayed on the display of the imaging device 20, and the type of thumbnail to be acquired is received in association with the correction process for the main image MP. In the type reception process, at least one of a developed thumbnail image, a color-corrected thumbnail image, and an edited thumbnail image is accepted as the type to be acquired. Note that the type reception process may be performed, for example, after the imaging device 20 is started, or may be performed immediately after the imaging process or each correction process described below.

[0097] Next, when the release button of the imaging device 20 is pressed, the processor 11 performs an imaging process at that time (S002). During the imaging process, the various components of the imaging device 20 are controlled to capture an image of a subject within the imaging range and obtain a main image MP. The main image MP obtained during the imaging process is an undeveloped RAW image. Then, an image file containing the main image MP is created upon obtaining the main image MP.

[0098] Next, the processor 11 executes a development correction step, in which the RAW image obtained in the imaging step is subjected to development correction processing (S003), thereby obtaining a developed image as the main image MP.

[0099] If the processor 11 has received "developed thumbnail image" as the type of thumbnail image to be acquired in the type receiving step (S004), the processor 11 performs an acquisition step in conjunction with the development correction step (S005). In this acquisition step, a developed thumbnail image is generated based on all or part of the developed image using the functions of the imaging device 20. This results in the acquired developed thumbnail image being stored in the header area of ​​the image file as additional information for the main image MP. The acquired developed thumbnail image is the first thumbnail image, and is stored in the image file in a manner that does not change in response to changes in the main image MP, even if the main image MP is subsequently corrected. In the acquisition step, a second thumbnail image that changes in response to changes in the main image MP may also be acquired as the developed thumbnail image, along with the first thumbnail image.

[0100] Next, when the user performs a color correction operation on the main image MP (S006), the processor 11 performs a color correction process in response to the user's operation (S007). In this process, a color correction process is performed on the developed image, thereby obtaining a color-corrected image as the main image MP. Accordingly, the main image MP in the image file is replaced from the developed image with the color-corrected image.

[0101] Furthermore, if the processor 11 has accepted "color-corrected thumbnail image" as the type of thumbnail image to be acquired in the type accepting step (S008), the processor 11 executes an acquisition step in conjunction with the color correction step (S009). In this acquisition step, a color-corrected thumbnail image is generated based on all or part of the color-corrected image using the functions of the imaging device 20. This results in the color-corrected thumbnail image being acquired and stored in the image file as additional information for the main image MP. The acquired color-corrected thumbnail image is a first thumbnail image, and is stored in the image file in a manner that does not change in response to changes in the main image MP, even if the main image MP is subsequently corrected. In the acquisition step, a second thumbnail image that changes in conjunction with changes in the main image MP may also be acquired as the color-corrected thumbnail image, along with the first thumbnail image.

[0102] Next, when the user performs an operation for processing or correcting the main image MP (S010), the processor 11 performs a processing or correction step in response to the user operation (S011). In this step, a processing or correction process is performed on the main image MP (more specifically, the developed image or the color-corrected image), thereby obtaining a processed image as the main image MP. Accordingly, the main image MP in the image file is replaced from the developed image or the color-corrected image with the processed image.

[0103] In addition, if the processor 11 has accepted "edited thumbnail image" as the type of thumbnail image to be acquired in the aforementioned type acceptance process (S012), it carries out a determination process and a selection acquisition process in conjunction with the execution of the processing correction process (S013, S014).

[0104] In the determination step, the processor 11 identifies the correction content of the main image MP in the processing and correction step, and specifically, identifies the amount of correction (the degree of change in the main image MP due to the correction) by comparing the main image MP before and after the correction. Then, the processor 11 determines whether or not to perform the acquisition step of acquiring a processed thumbnail image based on the identified amount of correction. Specifically, if the amount of correction is equal to or greater than a reference value, the processor 11 determines to perform the acquisition step of acquiring a processed thumbnail image.

[0105] 7 on the display of the imaging device 20 and acquires the user's selection as to whether or not to acquire an edited thumbnail image. If the user selects to acquire an edited thumbnail image, the processor 11 executes the edited thumbnail image acquisition step in accordance with the selection.

[0106] In the flow shown in FIG. 12B, both the determination step and the selection and acquisition step are performed, but this is not limitative, and only one of the determination step or the selection and acquisition step may be performed.

[0107] When acquiring an edited thumbnail image based on the results of the aforementioned determination step and selection / acquisition step (S015), the processor 11 executes an acquisition step (S016). In this acquisition step, an edited thumbnail image is generated based on all or part of the edited image using the functions of the imaging device 20. As a result, the edited thumbnail image is acquired and stored in the image file as supplementary information for the main image MP. The acquired edited thumbnail image is a first thumbnail image, and is stored in the image file in a manner that does not change in response to changes in the main image MP, even if the main image MP is subsequently corrected. In the acquisition step, a second thumbnail image that changes in response to changes in the main image MP may also be acquired as the edited thumbnail image, along with the first thumbnail image.

[0108] The image file creation flow ends when the above series of steps are completed. As explained above, in the image file creation flow of the first embodiment, the main image MP included in the image file is changed by the correction process. Furthermore, when the correction process is executed, if the correction process corresponds to the type of thumbnail image accepted in advance, a thumbnail image is acquired based on that type. Specifically, a thumbnail image TP is generated based on all or part of the corrected main image MP.

[0109] Furthermore, the thumbnail images TP acquired in the first embodiment include at least a first thumbnail image. Even if a correction process is performed on the main image after its generation, the first thumbnail image does not change in response to the transformation of the main image caused by the correction process. When the original main image is changed by the correction process, such a first thumbnail image can be used as evidence to prove what the main image looked like before the correction. In other words, in the first embodiment, it is possible to acquire thumbnail images that are highly reliable and useful as supplementary information for the main image.

[0110] [Second Embodiment] An image file creation device 10X according to the second embodiment has a function of executing a compositing process as a correction process for a main image MP. In the compositing process, two or more images including the main image MP are combined to obtain a new main image MP. The image before compositing will be referred to as a "basic image" below.

[0111] In the second embodiment, the following synthesis processing modes are selectable: a mode for reproducing multiple exposures, a mode for achieving super-resolution, a mode for achieving HDR (High Dynamic Range), and a mode for collage. In other words, in the second embodiment, when generating a composite image by synthesizing multiple basic images, the user selects one of multiple image generation modes, and the user's selection is accepted. Each synthesis processing mode will be described below.

[0112] (HDR Combining Processing Mode) In the HDR combining processing mode (hereinafter referred to as the first mode), a composite image is generated by combining a plurality of base images captured under different imaging conditions of the imaging device 20. Here, the imaging condition is the exposure amount, and in the first mode, a composite image is generated by combining a plurality of base images captured under different exposure amounts.

[0113] More specifically, one of the basic images used in the first mode is an image captured at an exposure (overexposure) greater than the standard exposure (the exposure set to the appropriate value at that time). The other basic images are images captured at an exposure (underexposure) less than the standard exposure. Then, as shown in FIG. 13A , the overexposed basic image and the underexposed basic image are combined by stitching together the properly exposed portions of each basic image. The combined image obtained in this manner is an image in which the bright and dark areas within the angle of view are properly represented (i.e., an image in which overexposure and underexposure are eliminated).

[0114] In addition, the first mode is not limited to using multiple basic images with different exposure levels, and multiple images with different imaging conditions other than exposure level, specifically ISO sensitivity, focus position at the time of imaging, and focal length, etc., may also be used as basic images.

[0115] (Super-resolution Combining Processing Mode) In the super-resolution combining processing mode (hereinafter referred to as the second mode), a plurality of basic images captured by shifting the imaging position of the imaging element in the imaging device 20 by one pixel or half pixel using the pixel shift function of the imaging device 20 are combined. The composite image obtained in this manner has a higher resolution than the basic images, as shown in FIG. 13B. Note that the time interval when capturing a plurality of basic images using the pixel shift function is, for example, about 1 / 100 seconds. Methods for creating a composite image using pixel shift are known, for example, from the methods described in Japanese Patent Application Laid-Open Nos. 2020-096301 and 2019-161564.

[0116] (Multiple Exposure Combining Processing Mode) In the multiple exposure combining processing mode (hereinafter referred to as the third mode), multiple basic images captured by the imaging device 20 at different locations are combined. For example, one of the multiple basic images is superimposed on the remaining basic image with its transparency set to a lower value than normal. The transparency is the degree to which the background is visible when the image is displayed, and can be freely adjusted within a range of 0 to 100% using the functions of the imaging device 20. The combined image obtained in this manner is an image in which the subjects of the multiple basic images captured at different locations are captured together, as shown in FIG. 13C .

[0117] (Collage Combining Processing Mode) In the collage combining processing mode (hereinafter referred to as the fourth mode), a plurality of basic images are combined in an arranged manner. Specifically, as shown in FIG. 13D , a composite image is obtained by arranging the basic images other than the template image among the plurality of basic images according to the image layout defined in the template image that serves as the background of the composite image. In the fourth mode, the basic images other than the template image are enlarged or reduced as appropriate according to the size of the template image. Furthermore, the plurality of basic images used in the fourth mode may be images selected by the user, or may be images automatically selected because the subject in the image satisfies predetermined conditions.

[0118] (Image File of Composite Image) In the second embodiment, when a composite image is generated by a compositing process, an image file is created in which the composite image is the main image MP. The header information of this image file stores additional information about the composite image, which is the main image MP. The additional information also includes a thumbnail image (ancillary image).

[0119] In the second embodiment, a thumbnail image TP is created based on at least one of the plurality of basic images used in the compositing process. Specifically, when a composite image is generated based on an image generation mode specified by a user, a thumbnail image TP is generated based on some or all of the plurality of basic images used to generate the composite image, depending on the image generation mode. This makes it possible to determine from the thumbnail image TP which basic images the composite image was generated from, i.e., which basic images were used as the material for the composite image. Note that the thumbnail image TP based on the basic images may be generated after the compositing process is performed, or may be generated before the compositing process is performed. Furthermore, a thumbnail image TP based on a composite image may be generated together with a thumbnail image TP based on the basic images.

[0120] Furthermore, the thumbnail image generated in the second embodiment is an image that does not change in response to changes in the composite image when the composite image is changed by color correction processing, processing correction processing, etc. By generating such a thumbnail image, it is possible to obtain a thumbnail image that is highly reliable and useful as incidental information related to the composite image. However, without being limited to this, the second embodiment may also generate a correctable thumbnail image, i.e., a thumbnail image (variable incidental image) that changes in response to changes in the composite image due to correction processing.

[0121] Next, functions of the image file creation device 10X according to the second embodiment will be described with reference to Fig. 14. The hardware configuration of the image file creation device 10X according to the second embodiment is the same as that of the first embodiment, and software such as an image file creation program is installed in the image file creation device 10X.

[0122] As shown in FIG. 14 , the image file creation device 10X according to the second embodiment includes a third generation unit 121, a reception unit 122, an image determination unit 123, a synthesis unit 124, a fourth generation unit 125, and a creation unit 126. These functional units are realized by cooperation between the hardware devices of the image file creation device 10X and software including an image file creation program. Some functions may also be realized using artificial intelligence (AI). Each functional unit will be described below.

[0123] (Third Generation Unit) The third generation unit 121 generates a basic image to be used in the compositing process. The procedure for generating a basic image is the same as the procedure for generating a developed image, a color-corrected image, or an edited image in the first embodiment, and therefore a description of the procedure for generating a basic image will be omitted. Furthermore, the third generation unit 121 may assign information to the basic image indicating whether the image is to be used to generate a thumbnail image (hereinafter, usability information). The usability information is stored, for example, as additional information in the image file of the basic image.

[0124] (Accepting Unit) When a composite image is generated by combining a plurality of basic images, the accepting unit 122 accepts an arbitrary image generation mode from a plurality of image generation modes. Specifically, the first to fourth modes described above are displayed as selectable modes on the mode designation screen of FIG. 15 . The user arbitrarily designates one of these modes, and the accepting unit 122 accepts the arbitrary image generation mode designated by the user. Note that the image generation mode may be accepted before the execution of the composition process, and may be accepted, for example, immediately after the imaging device 20 is started up.

[0125] (Image Determination Unit) When a composite image is generated by combining multiple basic images, the image determination unit 123 determines, from the multiple basic images, a basic image to be used to generate a thumbnail image. The basic image used to generate the thumbnail image (hereinafter referred to as a "generated basic image" as needed) may be determined by the image determination unit 123 by accepting a user's designation of the generated basic image, or may be determined automatically by the image determination unit 123. In the former case, for example, the multiple basic images used to generate the composite image are displayed so that they can be designated on the image designation screen of FIG. 16 . The user designates at least one image from the multiple basic images, and the image determination unit 123 accepts the user's designation of the generated basic image. The designation of the generated basic image may be accepted before the thumbnail image is generated, and may be accepted immediately after the execution of the composition process, for example.

[0126] When the image determination unit 123 automatically determines the generating base image, the generating base image may be determined in advance for each mode. For example, in the first mode, it may be determined that both an overexposed image and an underexposed image are to be the generating base image. In the second mode, it may be determined that the first image in time order is to be the generating base image. In the third mode, it may be determined that all of the images that are the source of synthesis are to be the generating base image. In the fourth mode, it may be determined that the image with the highest score as a result of subject analysis of the synthesized image, as described below, is to be the generating base image.

[0127] (Synthesis Unit) The synthesis unit 124 executes synthesis processing based on the image generation mode accepted by the acceptance unit 122, and generates a composite image by synthesizing a plurality of basic images in the synthesis processing. When executing synthesis processing in the third mode or the fourth mode, the synthesis unit 124 accepts designation of a plurality of basic images to be used in the synthesis processing from the user, and generates a composite image using the designated plurality of basic images.

[0128] (Fourth Generation Unit) The fourth generation unit 125 generates a thumbnail image as supplementary information of the composite image in conjunction with the synthesis process. In the second embodiment, the fourth generation unit 125 generates a thumbnail image based on at least one of the plurality of basic images in accordance with the image generation mode accepted by the acceptance unit 122.

[0129] Specifically, when the receiving unit 122 receives the first mode, the fourth generation unit 125 generates one or more thumbnail images based on some or all of the multiple generation basis images. More specifically, the fourth generation unit 125 generates two or more thumbnail images based on at least two of the multiple generation basis images with different exposure amounts during capture, as shown in FIG. 17 . The two or more generation basis images from which thumbnail images are generated may include, for example, an overexposed generation basis image and an underexposed generation basis image. In this case, images that have a significant impact on the composite image, such as overexposed or underexposed images, are automatically selected from the multiple base images used in the first mode. Then, thumbnail images can be generated based on the selected multiple generation basis images. Note that the above case is not limited to this; thumbnail images may also be generated based on generation basis images with a standard exposure amount (i.e., zero exposure). Alternatively, all of the multiple base images may be used as generation basis images, and thumbnail images may be generated based on each generation basis image. Furthermore, thumbnail images may also be generated based on either overexposed generation basis images or underexposed generation basis images.

[0130] Furthermore, when the receiving unit 122 receives the first mode, the fourth generating unit 125 may generate a thumbnail image based on each of the plurality of generated basic images for each of the plurality of generated basic images. In this case, a thumbnail image based on the generated basic image can be obtained for each of the plurality of generated basic images involved in the generation of the composite image. Note that in a configuration in which a thumbnail image is generated for each basic image, the plurality of basic images may include a basic image for which a thumbnail image is not generated.

[0131] When the receiving unit 122 receives the second mode, the fourth generation unit 125 generates a thumbnail image based on a portion of the plurality of basic images. Specifically, as shown in FIG. 18 , the fourth generation unit 125 generates a thumbnail image based on only a portion of the generated basic images (e.g., the first basic image in terms of time) among the plurality of basic images captured by shifting the imaging position using pixel shift. This is because, although the imaging positions of the plurality of basic images used in the second mode are shifted, most of the subjects in the images are the same. In other words, when performing a compositing process in the second mode, generating a thumbnail image based on only a portion of the basic images makes it possible to properly understand which basic images were used to generate the composite image.

[0132] When the receiving unit 122 receives the third mode, the fourth generation unit 125 generates thumbnail images based on all of the multiple basic images, as shown in FIG. 19 . That is, when multiple basic images captured at different locations are combined, the fourth generation unit 125 generates the same number of thumbnail images as the number of basic images based on each basic image. This is because the multiple basic images used in the third mode were captured at different locations, and therefore the subjects in the images are different. Therefore, when performing the combining process in the third mode, by generating thumbnail images based on all of the basic images, it is possible to properly understand which basic images were used to generate the combined image.

[0133] When the receiving unit 122 receives the fourth mode, the fourth generating unit 125 may generate a thumbnail image based on some of the basic images. Specifically, as shown in Fig. 20, the fourth generating unit 125 generates a thumbnail image based on only some of the generated basic images among the basic images arranged in a predetermined layout on a template image. Here, the basic images from which the thumbnail image is generated may be, for example, template images, i.e., images that indicate the layout of the basic images to be combined.

[0134] As another example of when the receiving unit 122 receives the fourth mode, the fourth generation unit 125 may select some of the basic images (strictly speaking, images other than the template image) from the multiple basic images based on the sizes of the multiple basic images in the composite image. In this case, the fourth generation unit 125 may generate thumbnail images based on the selected basic images. This makes it possible to obtain thumbnail images for basic images that have a large influence on the composite image, such as images that are relatively large in size.

[0135] As another example of when the receiving unit 122 receives the fourth mode, the fourth generation unit 125 may calculate a score for each of the multiple basic images in the composite image. The score may be calculated based on the resolution of the subject, such as blur or blur, in the basic images, the facial expression of the subject, the luminance (brightness) of each part of the subject, and the size of the basic images in the template image. The fourth generation unit 125 may then select some generated basic images (strictly speaking, images other than the template image) from the multiple basic images based on the calculated scores, and generate thumbnail images based on the selected generated basic images. This makes it possible to obtain thumbnail images for basic images that have a large influence on the composite image, such as images with a high score in the composite image.

[0136] Furthermore, when the receiving unit 122 receives the fourth mode, if the number of basic images used in the compositing process is equal to or less than a predetermined number, the fourth generating unit 125 may generate a thumbnail image based on all of the basic images. Note that the number of basic images that serves as a criterion for determining whether to generate a thumbnail image based on all of the basic images can be set to any number.

[0137] In the second embodiment, the fourth generation unit 125 may generate a thumbnail image based on at least one of the plurality of basic images and a thumbnail image based on a composite image. By generating thumbnail images for both the pre- and post-compositing images, it is possible to properly understand which basic images were used to generate which composite image. Note that, depending on the image generation mode, it may be clear that the main image MP is a composite image, in which case the basic images may be unnecessary. In such cases, it is sufficient to generate a thumbnail image based on the composite image (first thumbnail image), and a setting may be made not to generate thumbnail images based on the basic images.

[0138] In the second embodiment, the fourth generation unit 125 may generate a thumbnail image that does not change in response to changes in the composite image due to the correction process. This makes it possible to obtain a thumbnail image that is highly reliable and has high utility as incidental information related to the composite image.

[0139] Furthermore, depending on the image generation mode accepted by the accepting unit 122, the fourth generating unit 125 may generate a thumbnail image corresponding to the generation basic image determined by the image determining unit 123. For example, when the accepting unit 122 accepts the first mode or the second mode, the fourth generating unit 125 may generate a thumbnail image corresponding to the generation basic image determined by the image determining unit 123. Note that the thumbnail image corresponding to the generation basic image is a thumbnail image constructed by reducing all or part of the basic image.

[0140] Furthermore, which of the multiple basic images is used to generate a thumbnail image may be determined based on the availability information assigned to the basic image. In this case, the fourth generation unit 125 may generate the thumbnail image based on an image selected from the multiple basic images based on the availability information, i.e., the generation basic image used to generate the thumbnail image. The availability information is information attached to each basic image and indicates whether the basic image will be used as the generation basic image that will serve as the basis for the thumbnail image when used as the basis for a composite image. This makes it possible to appropriately determine whether a thumbnail image needs to be generated based on the availability information assigned to the basic image, and generate the thumbnail image that should be generated.

[0141] In the second embodiment, the creation unit 126 creates an image file of the composite image. The image file includes the composite image as the main image MP and additional information related to the composite image. The additional information includes the thumbnail image TP created by the fourth creation unit 125.

[0142] (Example of operation of image file creation device in second embodiment) Next, an image file creation flow using the image file creation device 10X in the second embodiment will be described. In the image file creation flow described below, the image file creation method of the present invention is used. Furthermore, the image file creation flow according to the second embodiment uses the image generation method of the present invention. In other words, the image file creation flow according to the second embodiment includes an image generation flow using the image generation method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order of execution of two steps in the flow may be reversed, as long as it does not deviate from the spirit of the present invention.

[0143] The steps in the image file creation flow according to the second embodiment are performed by the processor 11 included in the image file creation device 10X in the order shown in Fig. 21. That is, in each step in the image file creation flow, the processor 11 executes the processing corresponding to each step among the data processing defined by the image file creation program. The image file creation flow according to the second embodiment is a processing flow for generating a composite image by combining multiple basic images.

[0144] In the image file creation flow according to the second embodiment, first, the processor 11 performs a generation step A (S101). In the generation step A, a basic image to be used in a subsequent compositing step is generated. In addition, the generation step A is generally performed multiple times, and the imaging conditions (specifically, the exposure amount), imaging location, imaging position of the imaging element, etc. may be different between the generation steps A.

[0145] Next, the processor 11 performs a receiving step (S102). In the receiving step, the processor 11 displays a mode designation screen shown in Fig. 15 on the display of the imaging device 20 and accepts any image generation mode designated by the user from among a plurality of image generation modes. Note that the receiving step may be performed before the execution of the combining step, and may be performed, for example, after the imaging device 20 is started up.

[0146] Next, the processor 11 performs an image determination step, in which it determines a base image to be used to generate a thumbnail image from among the multiple base images to be synthesized (S103). Note that the image determination step may be omitted depending on the image generation mode accepted in the acceptance step.

[0147] The processor 11 then performs a compositing process (S104). In the compositing process, the processor 11 executes a compositing process according to the image generation mode accepted in the accepting process, and combines the plurality of base images to generate a composite image. Then, as the composite image is generated, the processor 11 sets the composite image as a main image MP and creates an image file including the main image MP.

[0148] Next, the processor 11 performs a generating step B (S105). This generating step B corresponds to the generating step in the second embodiment. In the generating step B, the processor 11 generates a thumbnail image TP based on at least one of the plurality of basic images used in the combining step, in accordance with the image generation mode accepted in the accepting step.

[0149] Specifically, if the first mode (HDR synthesis mode) is accepted in the accepting step, the processor 11 may generate one or more thumbnail images based on some of the plurality of basic images in the generating step B. More specifically, in the generating step B, the processor 11 may generate two or more thumbnail images based on at least two of the plurality of basic images that differ in the amount of exposure at the time of capture. In this case, it is preferable to generate thumbnail images corresponding to each basic image based on each of an overexposed basic image and an underexposed basic image (see FIG. 17 ).

[0150] As another example when the first mode is accepted in the accepting step, the processor 11 may generate thumbnail images based on all of the plurality of images in the generating step B. Specifically, in the generating step B, the processor 11 may generate thumbnail images based on each of the plurality of basic images for each basic image.

[0151] If the second mode (super-resolution synthesis mode) is accepted in the accepting step, the processor 11 may generate thumbnail images based on only some of the multiple basic images in the generating step B (see FIG. 18 ). In this case, the processor 11 may generate thumbnail images corresponding to the generated basic images determined in accordance with the user's designation accepted in the image determining step.

[0152] If the third mode (composite mode for multiple exposure) is accepted in the accepting step, the processor 11 may generate a thumbnail image based on all of the plurality of basic images in the generating step B (see FIG. 19).

[0153] If the fourth mode (composite mode for collage) is accepted in the accepting step, the processor 11 may generate a thumbnail image based on only a portion of the plurality of basic images in the generating step B (see FIG. 20 ). In this case, the processor 11 may generate a thumbnail image corresponding to the generated basic image determined in accordance with the user's specification accepted in the image determining step. Alternatively, the processor 11 may reference the availability information assigned to each basic image and generate a thumbnail image based on a generated basic image selected from the plurality of basic images based on the availability information. Alternatively, the processor 11 may automatically select a portion of the generated basic images from the plurality of basic images based on the size of each of the plurality of basic images in the composite image, and generate a thumbnail image based on the selected generated basic images. Alternatively, the processor 11 may calculate a score for each of the plurality of basic images based on the resolution of the subject in the basic image, and generate a thumbnail image based on a portion of the generated basic images automatically selected from the plurality of basic images based on the score.

[0154] The thumbnail image TP generated in the generation step B is stored in the image file of the composite image as supplementary information of the composite image. Furthermore, even if a correction process is performed on the composite image after its generation, the thumbnail image may be an image that does not change in response to the transformation of the composite image due to the correction process. In this case, the thumbnail image is highly reliable as evidence proving which basic images were used to generate the composite image, and is therefore a highly useful image.

[0155] In the generating step B, the processor 11 may generate a thumbnail image based on the composite image together with a thumbnail image based on at least one of the plurality of basic images. In this case, thumbnail images are generated for each of the images before and after compositing, so it is possible to properly grasp what basic images were used to generate what composite image.

[0156] When the above series of steps are completed, the image file creation flow according to the second embodiment ends. As described above, in the image file creation flow according to the second embodiment, thumbnail images are generated based on the basic images used in the compositing process. This makes it possible to accurately determine from the generated thumbnail images what basic images were used in the compositing process.

[0157] In the second embodiment, a thumbnail image is generated based on at least one of a plurality of basic images according to the image generation mode accepted in the accepting process. This allows the thumbnail image based on the basic image to be appropriately generated in a manner appropriate for the compositing process. Specifically, an image to be used as the original image for the thumbnail image is appropriately selected from the plurality of basic images according to the type of compositing process, and the thumbnail image is generated based on the selected basic image.

[0158] [Regarding the Third Embodiment] As in the above-described embodiments, the image file creation device 10Y according to the third embodiment creates an image file including a main image MP and a thumbnail image TP as supplementary information related to the main image MP. The image file creation device 10Y also has the function of analyzing the subject of the main image MP, determining conditions for generating thumbnail images based on the analysis results, and generating thumbnail images based on the determined conditions.

[0159] Specifically, in the third embodiment, the image file creation device 10Y analyzes the main subject among the subjects of the main image MP, and more specifically, analyzes the type, size, or resolution of the main subject, and quantitatively identifies these. The type refers to the classification (category) of the main subject. The size refers to the size of the area in the main image MP in which the main subject exists. The resolution is an index of the clarity of the main subject in the main image MP, and specifically refers to the degree of blur or blur. Methods for identifying these can include known subject detection technology, subject recognition technology, and image analysis technology.

[0160] Furthermore, if there are multiple subjects in the main image MP, the main subject is selected from the multiple subjects based on one or more of the following conditions i1 to i5: i1: Focus position (AF point) of the image capture device 20 in the main image MP; i2: The user's line of sight position in the main image MP, detected by a finder provided in the image capture device 20; i3: Characteristics of each subject in the main image MP; i4: The degree of change in the subject between the main image MP and a live view image captured earlier than the main image MP; i5: The type of subject specified by the user

[0161] When selecting a main subject based on conditions i1 and i2, for example, the subject closest to the gaze position detected by the AF point or the viewfinder of the imaging device 20 among multiple subjects can be selected as the main subject. Condition i3 includes, for example, the resolution (e.g., the degree of blur or blur) of the subject in the main image MP, image quality (e.g., clarity), or the facial expression of the subject. When selecting a main subject based on condition i3, for example, the main subject can be a subject whose degree of blur or blur is kept below a standard value or a subject with a smiling facial expression. Condition i4 includes the degree of change in the subject, which is the amount of change (degree of change) in the position or state of the subject from the time the live view image is captured to the time the main image MP is captured. When selecting a main subject based on condition i4, for example, the main subject can be a subject whose change in position or state is equal to or greater than a predetermined amount. When selecting a main subject based on condition i5, the main subject can be a subject that corresponds to a type specified by the user.

[0162] In the third embodiment, the main image MP to be analyzed includes a raw image before development. Furthermore, the image to be analyzed is not limited to the main image MP, but may be an image captured prior to the capture of the main image MP, i.e., a live view image. Because the live view image is similar to the main image MP, substantially similar analysis results can be obtained between the live view image and the main image MP.

[0163] In the third embodiment, the image file creation device 10Y determines image quality factors of thumbnail images or the angle of view of thumbnail images based on the analysis results of the main subject. The image quality factors are indices that determine the image quality of thumbnail images, and include quantified factors such as resolution, number of colors, number of gradations, dynamic range, and color gamut. The image file creation device 10Y then generates thumbnail images TP based on the determined image quality factors or angle of view.

[0164] As described above, in the third embodiment, the usefulness of thumbnail images can be improved. To explain this point with reference to Figure 22, a typical thumbnail image TP is created by reducing the entire main image MP, which is the original image, to the size of a typical thumbnail image (e.g., 400 x 320 pixels). If the subject in the main image MP is relatively small, creating a thumbnail image TP of that main image MP in the above manner will result in the subject being significantly smaller in the thumbnail image TP, as shown in Figure 22, making it difficult to identify.

[0165] In contrast, in the third embodiment, the image quality factor or angle of view of the thumbnail image TP is determined according to the type, size, or resolution of the subject in the main image MP. Then, as shown in FIG. 22 , the thumbnail image TP is generated based on the determined factor or angle of view. Specifically, the thumbnail image TP is generated so that its resolution or number of gradations is equal to or greater than a predetermined value (the value of a typical thumbnail image). For example, the thumbnail image TP is generated with a resolution of 1200 x 960 pixels. Alternatively, as shown in FIG. 22 , a subject area is extracted (trimmed) from the main image MP based on the determined angle of view, and the thumbnail image TP is generated based on the extracted subject area.

[0166] In the third embodiment, the thumbnail image TP can be generated in the above manner so that the area of ​​the subject in the thumbnail image TP is more clearly defined. As a result, the usability of the thumbnail image TP can be improved, and more specifically, the subject in the thumbnail image TP can be more clearly and favorably recognized.

[0167] An example of a use of the thumbnail images TP acquired in the third embodiment is to create training data for machine learning (AI learning) using the thumbnail images. The thumbnail images TP acquired in the third embodiment have a clearer subject than normal thumbnail images, and are therefore effective images as material for training data.

[0168] Furthermore, the thumbnail image TP generated in the third embodiment is preferably an image that does not change in response to changes in the main image MP caused by the correction process. Such a thumbnail image TP is a highly reliable image that serves as evidence proving what the main image MP looked like at the time of its generation. Therefore, even if corrections such as falsification have been made to the main image MP, the fact that such corrections have been made can be recognized by referring to the thumbnail image TP.

[0169] Furthermore, in the third embodiment, two types of thumbnail images TP based on the main image MP can be generated as shown in Fig. 23. The first thumbnail image TP is based on the entire main image MP, and more specifically, is a reduced image of the entire main image MP as shown in Fig. 23. Hereinafter, the first thumbnail image will be referred to as a "certificate thumbnail image." The certification thumbnail image is primarily used as evidence to prove what the main image MP looked like before correction when correction processing is performed on the main image MP.

[0170] The second thumbnail image is a thumbnail image TP based on the area of ​​the main subject in the main image MP, and more specifically, is an image obtained by reducing the area of ​​the main subject extracted (trimmed) from the main image MP, as shown in FIG. 23 . Hereinafter, the second thumbnail image will be referred to as a "learning thumbnail image." Learning thumbnail images are primarily used for creating training data for machine learning (AI learning). However, without being limited to this, a thumbnail image (proof thumbnail image) obtained by reducing the entire main image MP may also be used for creating training data for machine learning.

[0171] Next, functions of the image file creation device 10Y according to the third embodiment will be described with reference to Fig. 24. The hardware configuration of the image file creation device 10Y according to the third embodiment is the same as that of the first embodiment, and software such as an image file creation program is installed in the image file creation device 10Y.

[0172] As shown in FIG. 24 , the image file creation device 10Y according to the third embodiment includes a fifth generation unit 221, a selection unit 222, an analysis unit 223, a factor determination unit 224, a mode acceptance unit 225, a sixth generation unit 226, an encryption unit 227, and a creation unit 228. These functional units are realized by cooperation between the hardware devices of the image file creation device 10Y and software including an image file creation program. Some functions may also be realized using artificial intelligence (AI). Each functional unit will be described below.

[0173] (Fifth Generation Unit) The fifth generation unit 221 generates the main image MP. The procedure for generating the main image MP in the third embodiment is the same as the procedure for generating the main image MP (developed image, color-corrected image, or processed image) in the first embodiment, and therefore, the description thereof will be omitted.

[0174] (Selection Unit) When multiple subjects are present in the main image MP, the selection unit 222 selects a main subject from among the multiple subjects. Specifically, when the user does not specify the type of main subject, the selection unit 222 selects a main subject based on one or more of the above-mentioned conditions i1 to i4. Conditions i1 to i4 are contents that can be identified by analyzing the main image MP. Therefore, the selection unit 222 can automatically select a main subject from among the multiple subjects based on at least one of conditions i1 to i4.

[0175] On the other hand, if the type of main subject is specified by the user, the selection unit 222 selects the main subject based on one or more of the above-mentioned conditions i1 to i4 and the type specified by the user (i.e., condition i5). When the user specifies the type of main subject in this way, the selection unit 222 selects the main subject from among multiple subjects according to the specified type. When the user specifies the type of main subject, for example, a type specification screen shown in FIG. 25 may be displayed on the display of the imaging device 20. In this case, the type specification is accepted when the user specifies (inputs) the type of subject through the type specification screen.

[0176] (Analysis Unit) The analysis unit 223 performs an analysis process on the main image MP. In the analysis process, the type, size, or resolution of the subject in the main image MP is analyzed. Furthermore, when the selection unit 222 selects a main subject, the analysis process analyzes the type, size, or resolution of the selected main subject. Note that the target of the analysis process is not limited to the subject in the main image MP, but may also be a subject in a live view image captured prior to the capture of the main image MP.

[0177] (Factor etc. Determination Unit) The factor etc. determination unit 224 executes a determination process, and in this determination process, determines factors of the image quality of the thumbnail image TP or the angle of view of the thumbnail image TP (hereinafter, "factors etc.") based on the analysis results by the analysis unit 223. Specifically, the factor etc. determination unit 224 determines the resolution of the thumbnail image TP based on, for example, the size of the main subject analyzed by the analysis unit 223. More specifically, the factor etc. determination unit 224 determines the resolution of the thumbnail image TP so that the spatial frequency of the area of ​​the main subject in the thumbnail image TP is equal to or greater than a predetermined value. This makes it possible to generate the thumbnail image TP while ensuring sufficient clarity of the main subject in the thumbnail image. Note that the predetermined value related to the spatial frequency may be a value preset as an initial value, or may be a value that can be arbitrarily set and changed by the user.

[0178] Furthermore, if the size of the main subject analyzed by the analysis unit 223 is smaller than a predetermined size, the resolution or number of gradations of the thumbnail image TP may be increased in the determination process compared to when the size of the main subject is equal to or larger than the predetermined size. Alternatively, if the size of the main subject is smaller than the predetermined size, the angle of view of the thumbnail image TP may be reduced in the determination process compared to when the size of the main subject is equal to or larger than the predetermined size. Specifically, the angle of view of the thumbnail image TP may be set to be limited to the area of ​​the main subject in the main image MP. In this way, the resolution, number of gradations, or angle of view of the thumbnail image can be determined so that the main subject in the thumbnail image is more clearly visible and easier to recognize. Note that the thumbnail image TP generated based on the resolution, number of gradations, or angle of view determined as described above can be effectively used to create training data for machine learning.

[0179] (Mode Acceptance Unit) When a user selects a high-quality mode, the mode acceptance unit 225 accepts the selection of the high-quality mode. The high-quality mode is a mode that increases the image quality factors of a thumbnail image TP containing a main subject, specifically, the resolution or number of gradations, compared to the normal mode. For example, a user can select the high-quality mode when acquiring a high-quality thumbnail image for the purpose of creating training data for machine learning. The user's selection of the high-quality mode may be accepted by displaying a mode selection screen shown in FIG. 26 on the display of the imaging device 20. Note that the timing for accepting the selection of the high-quality mode is not particularly limited as long as it is before the generation of the thumbnail image TP. For example, the mode selection may be accepted immediately after capturing the main image MP (more specifically, the developed image).

[0180] When the mode receiving unit 225 receives the selection of the high-quality mode, the factor etc. determining unit 224 determines the resolution or number of gradations of the thumbnail image based on the selected high-quality mode and the analysis results by the analyzing unit 223. As a result, when the user selects the high-quality mode, the factors of the image quality of the thumbnail image TP can be determined in a way that reflects the user's desire for a high-quality thumbnail image.

[0181] (Sixth Generation Unit) The sixth generation unit 226 executes a thumbnail generation process to generate a thumbnail image TP based on the main image MP. In the thumbnail generation process, the sixth generation unit 226 generates a thumbnail image TP based on the main image MP based on the factors, etc. determined by the factor, etc. determination unit 224. In the third embodiment, the sixth generation unit 226 generates a certification thumbnail image and a learning thumbnail image as thumbnail images based on the main image MP. Note that the user may be able to arbitrarily specify whether to generate a certification thumbnail image or a learning thumbnail image, or whether to generate both thumbnail images. In addition, when a learning thumbnail image is generated, the resolution or number of gradations of the learning thumbnail image may be changeable in response to a user input operation.

[0182] Furthermore, when the mode receiving unit 225 receives the selection of the high-quality mode, the sixth generation unit 226 may generate a thumbnail image TP based on the region of the main subject in the main image MP, i.e., a learning thumbnail image. In this case, the thumbnail image TP is generated at the resolution or number of gradations determined by the factor determination unit 224. This allows the thumbnail image TP to be generated so that the main subject is more clearly defined, reflecting the intention of the user who selected the high-quality mode. Note that when the mode receiving unit 225 receives the selection of the high-quality mode, a thumbnail image TP based on the entire main image MP, i.e., a proof thumbnail image, may be generated at the resolution or number of gradations determined by the factor determination unit 224.

[0183] Furthermore, in the third embodiment, the sixth generation unit 226 may generate, as the thumbnail image TP, a thumbnail image that does not change in response to changes in the main image MP due to the correction process. However, without being limited to the above case, the sixth generation unit 226 may generate a thumbnail image (variable collateral image) that changes in response to changes in the main image MP due to the correction process.

[0184] (Encryption Unit) The encryption unit 227 acquires information obtained by encrypting the thumbnail image TP generated by the sixth generation unit 226. Specifically, when a certification thumbnail image is generated as the thumbnail image, the encryption unit 227 acquires information obtained by encrypting the certification thumbnail image. Furthermore, when a learning thumbnail image is generated as the thumbnail image, the encryption unit 227 acquires information obtained by encrypting the learning thumbnail image. The information obtained by encrypting the thumbnail image is, for example, a hash value obtained by hashing the thumbnail image TP.

[0185] The information acquired by the encryption unit 227 is stored in the image file as incidental information related to the main image MP. By storing the encrypted information of the thumbnail image TP in the image file in this manner, it is possible to detect tampering with the thumbnail image TP. For example, if a third party tampers with the thumbnail image TP, a pre-stored hash value Ha of the thumbnail image before tampering is compared with a hash value Hb obtained by hashing the thumbnail image after tampering. In this case, the hash value Ha is different from the hash value Hb, so it is possible to detect that the thumbnail image TP has been tampered with. Note that the encrypted information may be information in which the hash value is further encrypted using a private key. Furthermore, it is sufficient that at least the area of ​​the main subject of the thumbnail image TP can be encrypted; other areas do not need to be encrypted (in other words, they may be tampered with).

[0186] (Creation Unit) The creation unit 228 creates an image file including the main image MP generated by the fifth generation unit 221. The image file also includes additional information related to the main image MP. The additional information includes the thumbnail image TP generated by the sixth generation unit 226 and information acquired by the encryption unit 227.

[0187] (Example of operation of image file creation device in third embodiment) Next, an image file creation flow using image file creation device 10Y in the third embodiment will be described. In the image file creation flow described below, the image file creation method according to the third embodiment is used. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order in which two steps in the flow are executed may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0188] The steps in the image file creation flow according to the third embodiment are performed by the processor 11 included in the image file creation device 10Y in the order shown in Fig. 27. That is, in each process in the image file creation flow, the processor 11 executes the processing corresponding to each process among the data processing defined by the image file creation program.

[0189] In the image file creation flow according to the third embodiment, the processor 11 first executes a main image generation process (S201). In the main image generation process, a main image MP is generated using the same procedure as in the first embodiment. Following the generation of the main image MP, an image file of the main image MP is created. Appropriate correction processes (development correction process, color correction process, and processing correction process) are also performed on the main image MP. The main image MP in the image file changes depending on the execution of the correction processes.

[0190] Next, processor 11 detects a subject in the main image MP, and if multiple subjects are present in the main image MP, a selection step is performed before the subsequent analysis step (S202, S203). In the selection step, processor 11 selects a main subject from the multiple subjects. More specifically, if the user does not specify a type of main subject, processor 11 selects a main subject based on one or more of the above-described conditions i1 to i4. On the other hand, if the user specifies a type of main subject through the type specification screen shown in FIG. 25, processor 11 selects a main subject based on one or more of the above-described conditions i1 to i4 and the type specified by the user (i.e., condition i5).

[0191] Next, the processor 11 performs an analysis step to analyze the type, size, or resolution of the main subject of the main image MP (S204). The analysis step is not limited to analyzing the main subject of the main image MP, but may also analyze the main subject of a live view image captured earlier than the capture of the main image.

[0192] After the analysis step is performed, a factor determination step is performed (S207). However, if the user selects a high-quality mode, a mode acceptance step is performed before the factor determination step is performed (S205, S206).

[0193] In the factor etc. determination step, the processor 11 determines factors etc. of the image quality of the thumbnail image to be generated in the subsequent thumbnail generation step based on the analysis results of the analysis step. More specifically, in the factor etc. determination step, the processor 11 determines the resolution of the thumbnail image TP so that the spatial frequency of the area of ​​the main subject in the thumbnail image is equal to or greater than a predetermined value.

[0194] Furthermore, if the size of the main subject analyzed in the analyzing step is smaller than a predetermined size, the factor etc. determining step may increase the resolution or number of gradations of the thumbnail image compared to when the size of the main subject is equal to or larger than the predetermined size. Alternatively, if the size of the main subject is smaller than the predetermined size, the angle of view of the thumbnail image may be reduced compared to when the size of the main subject is equal to or larger than the predetermined size.

[0195] Furthermore, if a high-quality mode is selected by the user, the factor determination step may determine the resolution or number of gradations of the thumbnail image based on the selected high-quality mode and the analysis results from the analysis step.

[0196] Next, the processor 11 performs a thumbnail generation process (S208). In the thumbnail generation process, the processor 11 generates a thumbnail image TP based on the factors, etc. determined in the factor determination process. For example, if the size of the main subject in the main image MP is smaller than a predetermined size, the processor 11 generates a thumbnail image TP with a higher resolution or number of gradations than when the size of the main subject in the main image MP is equal to or larger than the predetermined size. Alternatively, the processor 11 generates a thumbnail image TP with a smaller angle of view, specifically, an angle of view limited to the area of ​​the main subject, than when the size of the main subject in the main image MP is equal to or larger than the predetermined size.

[0197] In the thumbnail generating step, the processor 11 generates a thumbnail image TP that does not change in response to changes in the main image MP due to the correction process. In the thumbnail generating step, the processor 11 may also generate a proof thumbnail image and a learning thumbnail image based on the factors determined in the determining step. For example, if the user selects a high-quality mode, a learning thumbnail image may be generated.

[0198] Next, the processor 11 performs an encryption step (S209). In the encryption step, the processor 11 acquires information obtained by encrypting the thumbnail image TP generated in the thumbnail generation step, specifically the proof thumbnail image or the learning thumbnail image. Specifically, the processor 11 acquires a hash value obtained by hashing the thumbnail image TP. The thumbnail image TP generated in the thumbnail generation step and the information acquired in the encryption step are then stored in the image file as auxiliary information for the main image MP.

[0199] When the above series of steps are completed, the image file creation flow according to the third embodiment ends. As described above, the image file creation flow according to the third embodiment analyzes the type, size, or resolution of the subject of the main image MP (strictly speaking, the main subject). The analysis results are then reflected in the image quality factors or angle of view of the thumbnail image TP based on the main image MP. This allows the thumbnail image TP based on the main image MP to be appropriately generated, taking into account the type, size, or resolution of the main subject of the main image MP.

[0200] Specifically, it is possible to generate thumbnail images TP while ensuring the clarity of the main subject in the thumbnail images TP. As a result, it is possible to obtain thumbnail images TP with improved usability. In particular, by making the main subject in the thumbnail images TP clearer, the thumbnail images TP can be effectively used, for example, as data for creating training data for machine learning aimed at image recognition.

[0201] [Regarding the Fourth Embodiment] As with the above-described embodiments, an image file creation device 10Z according to the fourth embodiment creates an image file including a main image MP and thumbnail images TP as supplementary information related to the main image MP. In the fourth embodiment, the main image MP is an image generated by pressing the release button of the imaging device 20. The image file creation device 10Z also has a function of generating multiple thumbnail images TP based on the main image MP and images captured before or after capturing the main image MP (specifically, live view images or recorded view images).

[0202] Specifically, in the fourth embodiment, a thumbnail image TP is generated based on the main image MP, and also based on at least one of a live-view image and a recorded-view image (see FIG. 29 ). This allows a thumbnail image TP to be generated based on an image captured temporally before or after the main image MP is captured, and the thumbnail image TP can be stored in an image file. Hereinafter, a thumbnail image TP generated based on the main image MP will be referred to as a "main thumbnail image." Furthermore, a thumbnail image TP generated based on a live-view image will be referred to as a "pre-capture thumbnail image," and a thumbnail image TP generated based on a live-view image or a recorded-view image will be referred to as a "post-capture thumbnail image."

[0203] In the fourth embodiment, the main image MP includes a RAW image, and as in the first embodiment, changes as a result of correction processing being performed on it, specifically, transitioning to one of a developed image, a color-corrected image, or an edited image as a result of the correction processing. Also, in the fourth embodiment, in conjunction with the correction processing on the main image MP, correction processing with the same content as the correction processing on the main image MP can be performed on a live view image or a quick view image.

[0204] In the fourth embodiment, the main thumbnail image, the before-capture thumbnail image, and the after-capture thumbnail image may be images that do not change in response to changes in the main image MP due to correction processing. In this case, whether or not correction processing has been performed on the main image MP can be determined from the above-described thumbnail images TP.

[0205] In the fourth embodiment, there is no particular limitation on the time point at which a live view image is used to generate a pre-capture thumbnail image. Similarly, there is no particular limitation on the time point at which a live view image is used to generate a post-capture thumbnail image. Furthermore, the number of pre-capture thumbnail images and the number of post-capture thumbnail images to be generated may each be determined arbitrarily.

[0206] In the following, as an example of the configuration of the fourth embodiment, a case will be described in which multiple thumbnail images TP are generated based on each of the main image MP, a live-view image, and a recorded-view image. However, this is not limited to this, and a single thumbnail image may be generated based on the main image MP and a live-view image or a recorded-view image. In this case, the thumbnail image may be, for example, a single image obtained by arranging a reduced image of the main image MP and a reduced image of the live-view image or a recorded-view image side by side.

[0207] Next, functions of the image file creation device 10Z according to the fourth embodiment will be described with reference to Fig. 28. The hardware configuration of the image file creation device 10Z according to the fourth embodiment is the same as that of the first embodiment, and software such as an image file creation program is installed in the image file creation device 10Z.

[0208] As shown in FIG. 28 , the image file creation device 10Z according to the fourth embodiment includes a first acquisition unit 321, a second acquisition unit 322, a third acquisition unit 323, a correction unit 324, a setting acceptance unit 325, a seventh generation unit 326, and a creation unit 327. These functional units are realized by cooperation between the hardware devices of the image file creation device 10Z and software including an image file creation program. Some functions may also be realized using artificial intelligence (AI). Each functional unit will be described below.

[0209] (First Acquisition Unit) The first acquisition unit 321 acquires the captured main image MP. To explain in detail, when the user fully presses the release button of the imaging device 20, the first acquisition unit 321 captures an image of a subject present within the imaging range of the imaging device 20 to acquire a RAW image, and develops the RAW image to acquire a developed image as the main image MP.

[0210] (Second Acquisition Unit) The second acquisition unit 322 acquires live view images captured before capturing the main image MP. More specifically, the user presses the release button halfway before fully pressing it. While the release button is being half-pressed, the second acquisition unit 322 continuously captures live view images at predetermined time intervals.

[0211] In the fourth embodiment, of the captured live-view images, those that satisfy a predetermined condition are temporarily stored in the memory 12 or the storage 16 as images captured before capturing the main image MP. Here, the predetermined condition is, for example, a condition that the state or position of the subject in the live-view image has changed by a predetermined amount or more from the image previously saved in the memory 12 or the storage 16 (the live-view image saved immediately before).

[0212] In other words, if the state or position of the subject has changed by a predetermined amount or more since the previously saved live-view image, the live-view image captured at that time is newly saved in the memory 12 or storage 16. In this case, the image saved in the memory 12 or storage may be replaced (updated) with a new live-view image that satisfies the above-described predetermined conditions. As described above, in the fourth embodiment, the live-view images temporarily saved in the memory 12 or storage 16 are limited to images that satisfy the predetermined conditions. This allows the live-view image used to generate the thumbnail image TP to be appropriately selected; more specifically, it allows the selection of a live-view image with a composition relatively close to that of the main image MP. Note that the present invention is not limited to the above case, and all images captured in live-view mode may be saved in the memory 12 or storage 16 regardless of whether the predetermined conditions are met.

[0213] (Third Acquisition Unit) The third acquisition unit 323 acquires a quickview image or a liveview image captured after capturing the main image MP. To explain in more detail, the third acquisition unit 323 stores, in the memory 12 or the storage 16, the quickview image or the liveview image captured a predetermined time after the user fully presses the release button. Then, a post-capture thumbnail image is generated (acquired) based on the stored image.

[0214] (Correction Unit) The correction unit 324 performs a correction process on the main image MP. Specifically, similar to the correction unit 23 of the first embodiment, the correction unit 324 performs a color correction process on the developed image as the main image MP, and also performs a processing correction process on the developed image as the main image MP or the color-corrected image.

[0215] Furthermore, in the fourth embodiment, the correction unit 324 can execute the same correction process as the correction process for the main image MP on the pre-captured thumbnail image or the post-captured thumbnail image in conjunction with the execution of the correction process for the main image MP. For example, when color correction is executed on the main image MP, the correction unit 324 can execute the same color correction process on the pre-captured thumbnail image or the post-captured thumbnail image. In this way, when the main image MP is corrected, by executing the same correction process on the pre-captured thumbnail image or the post-captured thumbnail image, these images can be changed to match the corrected main image MP.

[0216] (Settings Receiving Unit) When a user sets a setting to increase the resolution of each of the plurality of thumbnail images generated by the seventh generating unit 326, the settings receiving unit 325 receives the setting. Specifically, in the fourth embodiment, when thumbnail images are generated, a mode receiving screen shown in Fig. 30 is displayed on the display of the imaging device 20. The user selects either the normal mode or the high-resolution mode through the mode receiving screen.

[0217] When the user selects the high-resolution mode, the setting acceptance unit 325 accepts the selection as a setting to increase the resolution of the thumbnail images. Here, when the high-resolution mode is selected, the resolution of each of the multiple thumbnail images is increased compared to the normal mode. This allows the main thumbnail image, the pre-capture thumbnail image, and the post-capture thumbnail image to be generated more clearly (distinctly), thereby increasing the usefulness of each thumbnail image TP. In particular, when each of the multiple thumbnail images is used to create training data for machine learning, the setting to increase the resolution of each thumbnail image as described above becomes more meaningful. In the normal mode, the resolution of each of the multiple thumbnail images is set to the resolution of a typical thumbnail image.

[0218] Furthermore, when the high-resolution mode is selected, the resolution of each of the multiple thumbnail images does not need to be increased compared to the normal mode. Of the multiple thumbnail images, the resolution of at least the main thumbnail image may be increased, and the resolutions of the other thumbnail images may be increased in accordance with a user instruction. Furthermore, taking into account the capacity of the thumbnail images, the setting receiving unit 325 may receive a setting to increase the resolution of only the main thumbnail image among the multiple thumbnail images.

[0219] (Seventh Generation Unit) The seventh generation unit 326 executes processing to generate thumbnail images based on the main image MP, a live-view image, or a recording-view image. In the fourth embodiment, the processing to generate thumbnail images includes a first generation processing and a second generation processing. In the first generation processing, as shown in FIG. 29 , a main thumbnail image, a pre-captured thumbnail image, and a post-captured thumbnail image are generated based on the main image MP, a live-view image, and a recording-view image, respectively. The live-view image that is the original image of the pre-captured thumbnail image, and the recording-view image or live-view image that is the original image of the post-captured thumbnail image are each temporarily stored in the memory 12 or the storage 16 of the imaging device 20 and are read out when generating thumbnail images.

[0220] Furthermore, when a correction process is performed on the main image MP after the generation of each thumbnail image TP in the first generation process, the thumbnail images TP do not change in response to changes in the main image MP due to the correction process. Note that the seventh generation unit 326 may generate a main thumbnail image and generate either a pre-captured thumbnail image or a post-captured thumbnail image.

[0221] Furthermore, when the setting receiving unit 325 receives a setting to increase the resolution of the thumbnail images TP, the first generation process generates each thumbnail image TP at a resolution corresponding to the setting (i.e., a resolution higher than that in the normal mode). This generates each thumbnail image TP at a higher resolution, thereby improving the image quality and usability of each thumbnail image TP. Note that the above setting increases the resolution of at least the main thumbnail image, and therefore, in the first generation process when the above setting is received, at least the main thumbnail image is generated at a resolution corresponding to the above setting.

[0222] The second creation process generates a variable thumbnail image as a variable incidental image based on the main image MP. When a correction process is performed on the main image MP after its generation, the variable thumbnail image changes in response to changes in the main image MP due to the correction process. This variable thumbnail image is stored in an image file together with the main thumbnail image, which does not change in response to changes in the main image MP, so that the correction details of the main image MP can be understood by comparing these images.

[0223] Furthermore, if a correction process has been performed on the main image MP, the seventh generation unit 326 may separately perform a third generation process. In the third generation process, as shown in FIG. 31 , a thumbnail image based on the main image MP that has been corrected and a thumbnail image based on a live-view image or a recorded-view image that has been corrected in the same way as the main image MP are generated. Generating a thumbnail image based on the corrected main image MP and a thumbnail image based on a live-view image or a recorded-view image that has also been corrected in the same way improves the security of the thumbnail images. The security of thumbnail images will be described in detail later.

[0224] The content of the third generation process is not limited to the above. For example, in order to save data volume, the third generation process may generate only thumbnail images based on the main image MP that has been corrected. In other words, it is not necessary to generate thumbnail images based on live-view images or recorded-view images that have been corrected in the same way as the main image MP.

[0225] (Creation Unit) The creation unit 327 creates an image file of the main image MP acquired by the first acquisition unit 321. The image file stores supplementary information related to the main image MP, and the supplementary information includes thumbnail images TP generated by the seventh generation unit 326. The thumbnail images TP stored in the image file include a main thumbnail image based on the main image MP, a pre-capture thumbnail image based on a live view image, and a post-capture thumbnail image based on a recording view image or a live view image. That is, the image file created in the fourth embodiment includes multiple thumbnail images TP generated based on the main image, the live view image, and the recording view image, as shown in FIG.

[0226] (Example of operation of image file creation device in fourth embodiment) Next, an image file creation flow using image file creation device 10Z in the fourth embodiment will be described. In the image file creation flow described below, the image file creation method according to the fourth embodiment is used. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order in which two steps in the flow are executed may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0227] The steps in the image file creation flow according to the fourth embodiment are performed by the processor 11 included in the image file creation device 10Z in the order shown in Fig. 33. That is, in each process in the image file creation flow, the processor 11 executes the processing corresponding to each process among the data processing defined by the image file creation program.

[0228] In the image file creation flow according to the fourth embodiment, first, the user half-presses the release button before capturing the main image MP, which triggers the processor 11 to perform a pre-capture process (S301). In the pre-capture process, image capture is performed in live view mode, and live view images are continuously generated (acquired). During the pre-capture process, live view images that satisfy a predetermined condition are temporarily stored in the memory 12 or storage 16 as images captured before capturing the main image. The predetermined condition is that the state or position of the subject in the live view image has changed by a predetermined amount or more compared to the previously stored image (i.e., the live view image stored in the memory 12 or storage 16 immediately before).

[0229] When the user fully presses the release button to capture the main image MP, the processor 11 performs an image capturing process (S302). In the image capturing process, the processor 11 acquires a RAW image and develops the RAW image to obtain a developed image as the main image MP.

[0230] Thereafter, the processor 11 performs a post-imaging process when a predetermined time has elapsed since the main image MP was captured (S303). In the post-imaging process, the processor 11 acquires a live-view image after the main image MP has been captured, and the acquired live-view image is stored in the memory 12 or the storage 16. Note that the processor 11 may also acquire an image captured after the main image MP has been captured, i.e., a quick-view image.

[0231] After the post-imaging process is performed, a first generation process is performed (S306). However, if the user selects a high-resolution mode as a setting for increasing the resolution of the thumbnail images, a setting acceptance process is performed before the first generation process is performed (S304, S305). In the setting acceptance process, the processor 11 accepts a setting for increasing the resolution of at least the main thumbnail image among the multiple thumbnail images to be generated in the subsequent first generation process. More specifically, the processor 11 accepts the user's selection of the high-resolution mode as the above setting. Note that the setting acceptance process may accept a setting for increasing the resolution of each of the multiple thumbnail images, or may accept a setting for increasing the resolution of only the main thumbnail image.

[0232] In the first generation step, the processor 11 generates multiple thumbnail images TP based on the main image MP and the live-view image and recorded-view image stored in the memory 12 or the storage 16. Specifically, a main thumbnail image is generated based on the main image MP, a pre-captured thumbnail image is generated based on the live-view image, and a post-captured thumbnail image is generated based on the recorded-view image or the live-view image. Furthermore, each thumbnail image TP generated in the first generation step is an image that does not change in response to changes in the main image MP due to correction processing.

[0233] Furthermore, if the setting receiving step receives a user selection of a high resolution mode as a setting for increasing the resolution of the thumbnail images TP, the first generating step generates each thumbnail image TP at a resolution corresponding to the high resolution mode. Which of the multiple thumbnail images TP is generated at a resolution corresponding to the high resolution mode is determined based on the setting. However, at least the main thumbnail image is generated at a resolution corresponding to the high resolution mode.

[0234] The first generation process is not limited to a process of generating multiple thumbnail images TP, but may be a process of generating a single thumbnail image. Specifically, the main image MP, the live-view image, and the quick-view image may each be reduced in size, and a single thumbnail image TP may be generated by combining these images.

[0235] In the fourth embodiment, a second generation step is performed in addition to the first generation step (S307). In the second generation step, the processor 11 generates a variable thumbnail image based on the main image MP. The variable thumbnail image is an image (variable accessory image) that changes in response to changes in the main image MP due to the correction process. Note that the second generation step may be omitted.

[0236] After generating thumbnail images TP in the first and second generation steps, the processor 11 performs a creation step (S308). In the creation step, the processor creates an image file including the main image MP and the thumbnail image TP. If a dynamic thumbnail image is generated in the second generation step, an image file including the dynamic thumbnail image is created in the creation step (see FIG. 32).

[0237] After the image file is created, if a correction process is performed on the main image MP in the image file (S309), the processor 11 replaces (updates) the main image MP in the image file with the corrected main image MP. In addition, in conjunction with the execution of the correction process, the processor 11 performs a third generation step (S310). In the third generation step, the processor 11 generates a thumbnail image based on the main image MP that has been corrected, and a thumbnail image based on a live-view image or a recorded-view image that has been corrected in the same way as the main image MP. The thumbnail image generated in the third generation step is then added to the image file created in the creation step and stored in the image file.

[0238] The content of the third generation step is not limited to the above, and for example, it may be possible to generate only thumbnail images based on the main image MP that has been subjected to the correction process. In other words, it is not necessary to generate thumbnail images based on live-view images or quick-view images that have been corrected in the same way as the main image MP.

[0239] When the above series of steps are completed, the image file creation flow according to the fourth embodiment ends. As described above, in the image file creation flow according to the fourth embodiment, a thumbnail image TP is generated based on the main image MP and a live view image or a recorded view image, and is stored in the image file as supplementary information for the main image MP. This improves the security of the thumbnail image. Furthermore, because the image file includes multiple thumbnail images TP that are shifted in time, it becomes easier to perform machine learning on changes in the movement of a subject using a machine learning model.

[0240] Specifically, it is assumed that the thumbnail image TP itself may be tampered with. For example, if a thumbnail image TP based on the main image MP is tampered with, the credibility of the thumbnail image TP, which certifies the content of the main image MP, is undermined. On the other hand, in the fourth embodiment, as described above, thumbnail images TP are generated based on the main image MP and a live-view image or a recorded-view image. In this case, even if the thumbnail image TP based on the main image MP is tampered with alone, the tampering can be easily detected by referring to the thumbnail image TP based on the live-view image or the recorded-view image. Furthermore, to avoid the detection of the tampering, each thumbnail image TP would be changed (tampered with) in a similar manner, but this would increase the effort required to tamper with the thumbnail images TP and make the tampering more difficult. As a result, tampering of the thumbnail images TP can be suppressed, thereby improving security.

[0241] <<Other Embodiments>> The embodiments described above have been given to explain the present invention in an easy-to-understand manner, and are merely examples, and embodiments other than those described above may also be considered.

[0242] (Regarding the Timing of Thumbnail Image Creation and Image File Creation) In the above embodiment, when a main image MP is generated (acquired) or when a correction process is performed on the main image MP, a thumbnail image TP is created in conjunction with this, and an image file is also created. However, the timing of generating (acquiring) the thumbnail image TP and creating the image file is not particularly limited, and may be, for example, when a predetermined time has elapsed since the main image MP was acquired. Furthermore, the above timing may be determined arbitrarily by the user, and the thumbnail image may be generated and the image file may be created at a timing desired by the user.

[0243] (Regarding Types of Imaging Devices) In the above embodiment, the image generating device of the present invention is mounted on an imaging device. The imaging device is a digital camera that can be carried by a user, or an information processing terminal such as a smartphone or tablet terminal equipped with a camera function. However, the present invention is not limited to this, and the imaging device may be a fixed imaging device such as a security camera, or an imaging device having multiple imaging units (image sensors).

[0244] (Configuration of the Image Generation Device and Image File Creation Device) In the above embodiment, the imaging device that captures the main image MP generates supplementary information including a thumbnail image, and creates an image file that includes the main image MP and supplementary information. However, this is not limited to this, and the thumbnail image and image file may be created by a device other than the imaging device, specifically an information processing terminal such as a PC, smartphone, or tablet terminal connected to the imaging device. In this case, after the main image MP is captured by the imaging device, the main image MP is transmitted from the imaging device to the information processing terminal. Then, the information processing terminal that receives the main image MP performs image processing such as the correction processing and composition processing described above on the main image MP, and generates supplementary information including a thumbnail image TP.

[0245] (Regarding Processor Configuration) The processor included in the image generating device of the present invention includes various types of processors. The various types of processors include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various types of processors also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array). Furthermore, the various types of processors also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for performing specific processing.

[0246] Furthermore, one functional unit of the image generating device of the present invention may be configured by one of the various processors described above. Alternatively, one functional unit of the image generating device of the present invention may be configured by a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs, or a combination of an FPGA and a CPU. Furthermore, multiple functional units of the image generating device of the present invention may be configured by one of the various processors, or two or more of the multiple functional units may be combined into a single processor. Furthermore, as in the above-described embodiment, one processor may be configured by a combination of one or more CPUs and software, and this processor may function as multiple functional units.

[0247] Furthermore, a processor may be used that realizes the functions of the entire system including the multiple functional units in the image generating device of the present invention on a single IC (Integrated Circuit) chip, as typified by, for example, an SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements.

[0248] 10 Image file creation device (image generation device) 10X, 10Y, 10Z Image file creation device (image generation device) 11 Processor 12 Memory 13 Communication interface 14 Input device 15 Output device 16 Storage 20 Imaging device 21 Type reception unit 22 First generation unit 23 Correction unit 24 Determination unit 25 Selection acquisition unit 26 Second generation unit 27 Creation unit 121 Third generation unit 122 Reception unit 123 Image determination unit 124 Synthesis unit 125 Fourth generation unit 126 Creation unit 221 Fifth generation unit 222 Selection unit 223 Analysis unit 224 Factor etc. determination unit 225 Mode reception unit 226 Sixth generation unit 227 Encryption unit 228 Creation unit 321 First acquisition unit 322 Second acquisition unit 323 Third acquisition unit 324 Correction unit 325 Setting reception unit 326 Seventh generation unit 327 Creation unit AD Additional information MP Main image TP Thumbnail image (additional image)

Claims

1. a receiving step of receiving an arbitrary image generation mode from a plurality of image generation modes when generating a composite image by combining a plurality of basic images; a generating step of generating an auxiliary image for the basic image before synthesis as auxiliary information of the synthetic image based on at least one basic image among the plurality of basic images in accordance with the image generation mode accepted in the accepting step; An image generating method comprising:

2. An image determination step of determining, from among the plurality of basic images, the basic image to be used for generating the auxiliary image in accordance with a user's designation, The image generating method according to claim 1 , wherein in said generating step, said auxiliary image corresponding to said basic image determined in said image determining step is generated.

3. The image generating method according to claim 1 , wherein the auxiliary image is an image that does not change with a change in the composite image due to a correction process.

4. the plurality of image generation modes includes a first mode in which the plurality of basic images, which are captured under different imaging conditions of an imaging device at the time of capturing an image, are combined to generate the composite image; The image generating method according to claim 1 , wherein, when the first mode is received in the receiving step, the generating step generates one or more of the auxiliary images based on some or all of the plurality of basic images.

5. the imaging condition is an exposure amount, The image generating method according to claim 4, wherein, when the first mode is received in the receiving step, in the generating step, two or more of the auxiliary images are generated based on at least two or more of the plurality of basic images having different exposure amounts at the time of shooting.

6. The image generating method according to claim 4 , wherein, when the first mode is received in the receiving step, in the generating step, the auxiliary image based on each of the plurality of basic images is generated for each basic image.

7. the plurality of image generation modes includes a second mode in which the plurality of basic images captured by shifting an imaging position of an imaging element in an imaging device are synthesized to generate the composite image, The image generating method according to claim 1 , wherein, when the second mode is received in the receiving step, the generating step generates the auxiliary image based on a part of the plurality of basic images.

8. the plurality of image generation modes includes a third mode in which the plurality of basic images captured by an imaging device at different locations are combined to generate the composite image; The image generating method according to claim 1 , wherein, when the third mode is received in the receiving step, the generating step generates the auxiliary image based on all of the plurality of basic images.

9. the plurality of image generation modes includes a fourth mode in which the plurality of base images are synthesized in an arranged manner to generate the synthesized image; The image generating method according to claim 1 , wherein, when the fourth mode is received in the receiving step, the generating step generates the auxiliary image based on a part of the plurality of basic images.

10. The image generating method according to claim 9, wherein in the generating step, a portion of the basic images is selected from the plurality of basic images based on the sizes of the plurality of basic images in the composite image, and the auxiliary image is generated based on the selected portion of the basic images.

11. The image generating method according to claim 1 , wherein in the generating step, the auxiliary image based on at least one of the plurality of basic images and the auxiliary image based on the composite image are generated.

12. For each of the plurality of basic images, information indicating whether or not the basic image is an image to be used in generating the auxiliary image is stored; The image generating method according to claim 1 , wherein in the generating step, the auxiliary image is generated based on a basic image selected from the plurality of basic images based on the information.

13. 13. An image file creating method for creating an image file including a composite image as a main image and the accessory image created by the image creating method according to claim 1.

14. 1. An image generating device comprising a processor, The processor receives an arbitrary image generation mode from a plurality of image generation modes when generating a composite image by synthesizing a plurality of basic images; The processor generates an auxiliary image for the basic image before synthesis as auxiliary information of the synthetic image based on at least one basic image among the multiple basic images in accordance with the received image generation mode.