Image processing device, method for controlling the image processing device, and program

Abstract

To provide an image processing apparatus that can provide an image file after conversion that can use all pieces of annotation information recorded in an image file before conversion.SOLUTION: A digital camera 100 converts a HEIF image file into a JPEG image file according to an image conversion instruction issued by a user, and when annotation information included in the HEIF image file is not related image data, records the annotation information in an application marker in the JPEG image file. When the annotation information included in the HEIF image file is the related image data, the digital camera 100 records link information on the annotation information in the application marker in the JPEG image file.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, and a program.

Background Art

[0002] In recent years, A.I. technologies such as deep learning have been utilized in various technical fields. For example, in a digital camera having a function of detecting a human face from image data obtained by shooting, technologies for accurately detecting objects other than humans, such as animals, plants, buildings, and objects, using an inference model have been proposed (see, for example, Patent Document 1). Further, technologies for associating and recording information about a detected object as annotation information with image data have been proposed. Specifically, an image file in which annotation information and image data are recorded is generated. As annotation information, for example, information indicating a region of a detected object with a point, a rectangle, a circle, etc. is considered to be recorded. Further, it is also considered to cut out image data of a detected object as related image data of the original image data and record the cut-out image data in association with the original image data. Furthermore, it is also considered to record properties such as the color, shape, and type of the cut-out image data in association with the original image data.

[0003] On the other hand, digital cameras are equipped with a function of converting RAW image files or HEIF (High Efficiency Image File Format) image files into more playback-compatible JPEG image files.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] JPEG image files have a smaller storage capacity for annotation information compared to other image file formats such as HEIF. Therefore, when converting RAW or HEIF image files to JPEG, it's not possible to record all the annotation information from the original image file into the JPEG file. As a result, some of the annotation information from the original image file cannot be used in the converted image file.

[0006] The object of the present invention is to provide an image processing apparatus, a control method for the image processing apparatus, and a program that can provide a converted image file that makes available all annotation information recorded in the image file before conversion. [Means for solving the problem]

[0007] To solve the above problems, the image processing apparatus of the present invention includes a conversion means for converting an image file of a first format, which includes image data and annotation information of the image data, into an image file of a second format, the recording capacity of a predetermined area for recording annotation information is smaller than that of the first format image file, wherein the conversion means records the annotation information in the predetermined area of ​​the second format image file if the annotation information included in the first format image file is not predetermined annotation information that results in a data amount of more than a predetermined value, and records the link information of the annotation information in the predetermined area of ​​the second format image file if the annotation information included in the first format image file is predetermined annotation information. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a converted image file that can utilize all the annotation information recorded in the image file before conversion. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example configuration of a digital camera as an image processing device according to this embodiment. [Figure 2] This diagram illustrates the structure of the HEIF image file used in this embodiment. [Figure 3] Figure 1 is a flowchart showing the steps of the shooting control process performed in the digital camera. [Figure 4] This is a diagram illustrating the structure of a JPEG image file used in this embodiment. [Figure 5A] Figure 1 is a flowchart showing the procedure for the format conversion process performed in the digital camera. [Figure 5B] Figure 5A is a flowchart showing the procedure for the header creation process in step S506. [Figure 5C] Figure 5A is a flowchart showing the procedure for saving a JPEG image file in step S507. [Figure 6] This diagram illustrates the structure of an image file generated by converting an HEIF image file in this embodiment. [Figure 7] Figure 5B is a flowchart showing another step in the header creation process. [Figure 8] Figure 5C is a flowchart showing another procedure for saving a JPEG image file. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] In the following description, a digital camera that uses an inference model to classify subjects will be described as an example of an image processing device in this embodiment. However, the image processing device is not limited to a digital camera. For example, the image processing device can be any device that converts image files such as HEIF stored in a memory unit to other image file formats, such as a smartphone or tablet PC.

[0012] Furthermore, this embodiment describes a configuration in which an image file of the first format, an HEIF image file, is converted to an image file of the second format, a JPEG image file, but the configuration is not limited to this. The image file of the first format may be an image file of a format other than HEIF, as long as it includes image data and annotation information for said image data, and the recordable capacity of the predetermined area for recording the annotation information is greater than that of the image file of the second format. Also, the image file of the second format may be an image file of a format other than JPEG, as long as the recordable capacity of the predetermined area for recording the annotation information is smaller than that of the image file of the first format.

[0013] Figure 1 is a block diagram showing an example configuration of a digital camera 100 as an image processing device according to this embodiment.

[0014] Barrier 10 is a protective member that covers the imaging unit of the digital camera 100, including the imaging lens 11, to prevent dirt and damage to the imaging unit. The imaging lens 11 forms an optical image on the imaging surface of the image sensor 13. The shutter 12 has an aperture function. The image sensor 13 converts the optical image formed on the imaging surface into an electrical signal. The image sensor 13 is composed of, for example, a CCD or CMOS element. The A / D converter 15 converts the analog signal output from the image sensor 13 into a digital signal. In Figure 1, the A / D converter 15 is denoted as A / D15. The digital signal converted by the A / D converter 15 is written to memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on the shooting information, and these development parameters are also written to memory 25. The development parameters include parameters used for exposure control and parameters used for image processing such as white balance, color space, and contrast.

[0015] The timing generation unit 14 supplies clock signals and control signals to the image sensor 13, A / D converter 15, and D / A converter 21. In Figure 1, the D / A converter 21 is denoted as D / A21. The timing generation unit 14 is controlled by the memory control unit 22 and the system control unit 50A. The image processing unit 20 performs various image processing such as predetermined pixel interpolation, color conversion, correction, and resizing on the data from the A / D converter 15 or the data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and calculation processing using the captured image data and provides the obtained calculation results to the system control unit 50A. The system control unit 50A controls the exposure control unit 40 and the distance measurement control unit 41 based on the provided calculation results to realize AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing.

[0016] The image processing unit 20 performs predetermined arithmetic processing using the captured image data, and performs AWB (Auto White Balance) processing based on the obtained arithmetic result. Further, the image processing unit 20 performs compression processing on the image data stored in the memory 25 in a format such as JPEG, MPEG-4 AVC, HEVC (High Efficiency Video Coding). Furthermore, the image processing unit 20 performs decompression processing on the data compressed in these formats. The image processing unit 20 writes the processed data into the memory 25.

[0017] In addition, the image processing unit 20 performs predetermined arithmetic processing using the captured image data, and performs editing processing of various image data. Specifically, the image processing unit 20 performs trimming processing to adjust the display range and size of the image by making non-display unnecessary portions around the image data, and resizing processing to change the size by enlarging or reducing the image data or display elements of the screen. Furthermore, the image processing unit 20 performs RAW development to convert the data subjected to compression processing or decompression processing into JPEG image data after performing image processing such as color conversion. Also, the image processing unit 20 performs processing to extract a specified frame of a video format such as MPEG-4 and convert it into JPEG data.

[0018] In addition, the image processing unit 20 also performs processing such as superimposing OSD (On-Screen Display) such as menus and arbitrary characters to be displayed on the display unit 23 together with the display image data.

[0019] In addition, the image processing unit 20 performs processing to detect the subject area in the image data using the input image data, distance information from the subject obtained from the imaging element 13 at the time of shooting, and the like. Through this processing, information such as the position and size of the subject in the image data can be obtained.

[0020] The memory control unit 22 controls the A / D converter 15, the timing generation unit 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The RAW image data generated through the A / D converter 15 is written to the image display memory 24 and the memory 25 via the image processing unit 20 and the memory control unit 22, or via the memory control unit 22 without going through the image processing unit 20.

[0021] Image data for display written to the image display memory 24 is displayed on the display unit 23 via the D / A converter 21. By sequentially displaying the captured image data on the display unit 23, it is possible to realize an electronic viewfinder function that displays live video. Memory 25 stores captured still image data and video data. Memory 25 has sufficient storage capacity to store a predetermined number of still image data and a predetermined amount of video data. Memory 25 is also used as a workspace for the system control unit 50A.

[0022] The exposure control unit 40 controls the shutter 12, which has an aperture function. The exposure control unit 40 also works in conjunction with the flash 44 to realize a flash metering function. The distance measuring control unit 41 controls the focusing of the photographic lens 11. The zoom control unit 42 controls the zooming of the photographic lens 11. The barrier control unit 43 controls the operation of the barrier 10, which is a protective member. The flash 44 has an AF assist light projection function and a flash metering function.

[0023] The system control unit 50A controls the entire digital camera 100. The non-volatile memory 51 is a non-volatile memory that can electrically erase and record data. For example, an EEPROM is used for the non-volatile memory 51. In addition to the program, map information and other data are also stored in the non-volatile memory 51.

[0024] The shutter switch 61 (SW1) turns ON during the operation of the shutter button 60 and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) turns ON when the operation of the shutter button 60 is completed and instructs the start of a series of shooting operations including exposure processing, development processing, and recording processing. In the exposure processing, the signal read from the image sensor 13 is written to the memory 25 as RAW data via the A / D converter 15 and the memory control unit 22. In the development processing, the RAW data written to the memory 25 is developed using the calculation results from the image processing unit 20 and the memory control unit 22, and written to the memory 25 as image data. In the recording processing, the image data is read from the memory 25, the image data compressed by the image processing unit 20 is stored in the memory 25, and then the compressed image data is written to the external recording medium 91 via the card controller 90.

[0025] The control unit 63 is equipped with various buttons and a touch panel. For example, the control unit 63 includes a power button, a menu button, a mode switch for switching between shooting mode / playback mode / other special shooting modes, a directional pad, a set button, a macro button, and a multi-screen playback page change button. The control unit 63 also includes, for example, a flash setting button, a single-shot / continuous-shot / self-timer switching button, a menu navigation + (plus) button, a menu navigation - (minus) button, a shooting quality selection button, an exposure compensation button, and a date / time setting button.

[0026] The metadata generation and analysis unit 70 generates metadata in the image data format, such as Exif (Exchangeable image file format), based on the shooting information when recording image data to the external recording medium 91. The metadata includes, for example, shooting settings information, information about the image data, and characteristic information of the subject contained in the image data. The metadata generation and analysis unit 70 also analyzes the metadata attached to the image data read from the external recording medium 91. Furthermore, when recording video data, the metadata generation and analysis unit 70 can also generate metadata for each frame and attach the metadata for all frames to the video data.

[0027] The power supply 80 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The power control unit 81 supplies power from the power supply 80 to each part of the digital camera 100. The card controller 90 transmits and receives data with an external recording medium 91 such as a memory card. The external recording medium 91 is, for example, a memory card and records image data (still image data and video data) captured by the digital camera 100.

[0028] The inference model recording unit 72A stores the inference model acquired by the communication unit 71A from an external device 101 or the like. The inference engine 73A performs inference by inputting an image file containing image data acquired from the system control unit 50A and annotation information of said image data into the inference model stored in the inference model recording unit 72A. In this embodiment, the inference engine 73A may use the inference model acquired by the communication unit 71A from an external device 101 or the like and recorded in the inference model recording unit 72A, or it may use the inference model that has been retrained by the learning unit 74A.

[0029] The inference model consists of an input layer, a hidden layer (neurons), and an output layer. The input layer receives image files output from the system control unit 50A. The hidden layer consists of multiple layers. The number of layers in the hidden layer is determined as appropriate in the design, and the number of neurons in each layer is also determined as appropriate in the design. The output layer outputs annotation information corresponding to the image files input to the input layer.

[0030] In this embodiment, the inference model used by the inference engine 73A is assumed to be an inference model that infers the classification of subject areas detected from image data. This inference model is generated by deep learning using an external device 101 or the like, using image data of various subjects and the results of their classification (for example, classification of animals such as dogs and cats, or classification of subjects such as people, animals, plants, and buildings) as training data. The digital camera 100 may also acquire other inference models from the external device 101 or the like and use those other acquired inference models.

[0031] The learning unit 74A retrains the inference model stored in the inference model recording unit 72A according to instructions received from the system control unit 50A. The learning unit 74A has a training data recording unit 74a. The training data recording unit 74a records information about the training data used to retrain the inference model. The learning unit 74A retrains the inference model using the training data recorded in the training data recording unit 74a and stores the retrained inference model in the inference model recording unit 72A. As a result, the inference engine 73A can perform inference using the retrained inference model. In this embodiment, the inference model recording unit 72A maintains a management version for managing the update status of the inference model, and the update status of the inference model can be grasped based on this management version.

[0032] The communication unit 71A has a communication circuit for transmitting and receiving data. Communication by the communication unit 71A may be wireless communication such as Wi-Fi or Bluetooth, or wired communication such as Ethernet or USB. For example, the communication unit 71A can communicate with the communication unit 71B of the external device 101. The communication unit 71A functions as a communication unit that sends and receives various information, such as not only image files with annotation information created by the inference engine 73A, but also inference models and training data. The digital camera 100 can restrict the information it transmits depending on whether the external device 101 is associated with the camera or not.

[0033] The external device 101 includes a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system recording unit 50B, and a communication unit 71B. The learning unit 74B creates an inference model according to instructions received from the system recording unit 50B, etc. The inference model recording unit 72B records the inference model transmitted from the digital camera 100 and the inference model created by the learning unit 74B.

[0034] In this embodiment, the digital camera 100 detects objects such as people, animals, plants, buildings, and other objects from the captured image data. The digital camera 100 then generates an image file containing the captured image data and annotation information for that image data. As annotation information, the digital camera 100 records in the image file information that indicates the area of ​​the detected object as a point, rectangle, circle, etc. The digital camera 100 also extracts the image data of the detected object as related image data from the original image data and records the extracted image data (related image data) in an image file. Furthermore, the digital camera 100 records properties such as color, shape, and type of the extracted image data in the image file. In this embodiment, for example, when the user instructs the digital camera 100 to capture HEIF image data, the digital camera 100 generates the HEIF image file 200 shown in Figure 2.

[0035] Figure 2 is a diagram illustrating the structure of the HEIF image file 200 used in this embodiment. As shown in Figure 2(a), the HEIF image file 200 is composed of multiple regions called BOXes, and records data in a tree structure. Data may be recorded directly in a BOX, or other BOXes may be recorded as child elements. The HEIF image file 200 includes an 'ftype' BOX 201 for recording the type of image file, a 'meta' BOX 202 for recording metadata including annotation information, and an 'mdat' BOX 218 for recording image data. Note that Figure 2(a) shows only the minimum number of BOXes necessary for describing the present invention, but the HEIF image file 200 may include other BOXes besides those described above.

[0036] The 'meta'BOX202 has the following child elements: 'item definition'BOX203, 'item association'BOX207, 'item attribute definition'BOX211, and 'item location information'BOX215.

[0037] The 'Item Definition' BOX 203 contains the items to be defined as annotation information as child elements. In Figure 2(a), the 'Item Definition' BOX 203 contains the item definition 204 for the main image, the item definition 205 for the related image, and the item definition 206 for the region. If you want to define annotation information other than these, you add the items you want to define to the 'Item Definition' BOX 203. The item definition 204 for the main image contains the definition of the item number of the image data 250 in Figure 2(b), which is the main image data of the HEIF image file 200, and the definition that the item type is "image". The item definition 205 for the related image contains the definition of the item number of the related image data 251 in Figure 2(c), which is extracted from image data 250, and the definition that the item type is "image". The item definition 206 for a region records the definition of the item number of the region corresponding to the related image data 251 in the image data 250, for example, region 253 in Figure 2(d), and the definition that the item type is "region".

[0038] The 'Item Association' BOX 207 records item reference information as child elements. In Figure 2(a), the 'Item Association' BOX 207 records the relationship between the main image and the related image 208, the relationship between the main image and the main image's attribute information 209, and the relationship between the related image and the related image's attribute information 210. If you want to define annotation information other than these, the items you want to define are added to the 'Item Association' BOX 207. The relationship between the main image and the related image 208 records the item number of the main image data in association with the item number of the related image data. The relationship between the main image and the main image's attribute information 209 records the item number of the main image data in association with the main image data's attribute information 212, which will be described later. The relationship between the related image and the related image's attribute information 210 records the item number of the related image data in association with the related image data's attribute information 213, which will be described later.

[0039] The 'Item Attribute Definition' BOX 211 records property information such as the attribute information of the item as child elements. In Figure 2(a), the 'Item Attribute Definition' BOX 211 records the attribute information 212 of the main image, the attribute information 213 of the related image, and the attribute information 214 of the region. If you want to define annotation information other than these, you add the item you want to define to the 'Item Attribute Definition' BOX 211. The attribute information 212 of the main image records the size, color space information, pixel information, rotation information, etc. of the main image data. The attribute information 213 of the related image records the size, color space information, pixel information, rotation information, etc. of the related image data, and also records the attributes of the detected object. By recording the attributes of the detected object in the attribute information 213 of the related image, for example, as shown in Figure 2(d), the string "Flower" 254 indicating that attribute can be displayed in the region 253 corresponding to the related image data. The attribute information 214 of the region records the shape and coordinates of the region of the detected object.

[0040] The 'item location information' BOX 215 records the location information of the image data in 'mdat' BOX 218 as a child element. In Figure 2(a), the location information 216 of the main image and the location information 217 of the related image are recorded. The location information 216 of the main image records the location information of the main image data in 'mdat' BOX 218. The location information 217 of the related image records the location information of the related image data in 'mdat' BOX 218. 'mdat' BOX 218 records the main image data 219 and the related image data 220 of the main image data 219. Note that image data in formats other than HEIF can also be recorded in mdat' BOX 217.

[0041] Next, we will explain how to take a picture using the digital camera 100. Figure 3 is a flowchart showing the procedure for the shooting control process performed in the digital camera 100 shown in Figure 1. The shooting control process in Figure 3 is realized when the system control unit 50A executes a program stored in the non-volatile memory 51 or the like. The shooting control process in Figure 3 is executed when the user sets the shooting mode using the mode switching switch included in the operation unit 63.

[0042] In Figure 3, first, the system control unit 50A determines whether the user has pressed the shutter switch 61 (SW1) or shutter switch 62 (SW2) to instruct the capture of HEIF image data (step S301). If it is determined in step S301 that the user has instructed the capture of HEIF image data, the process proceeds to step S302. If it is determined in step S301 that the user has not instructed the capture of HEIF image data, the process proceeds to step S308, which will be described later.

[0043] In step S302, the system control unit 50A performs the shooting process. During the shooting process, the system control unit 50A controls the distance measurement control unit 41 and the exposure control unit 40 to perform AF processing and AE processing, and saves the video data output from the image sensor 13 to the memory 25. The system control unit 50A also controls the image processing unit 20 to generate HEIF image data by compressing the video data stored in the memory 25 using HEIF.

[0044] Next, in step S303, the system control unit 50A controls the image processing unit 20 to perform subject detection processing on the video data stored in the memory 25. This detects the subject area in the video data.

[0045] Next, in step S304, the system control unit 50A saves the HEIF image file 200, which includes the image data generated in step S302 and the subject area information detected in step S303, to the external recording medium 91. The subject area information is recorded in the 'meta'BOX 202 of the HEIF image file 200 by the metadata generation and analysis unit 70 as information used in the inference process described later.

[0046] Next, in step S305, the system control unit 50A determines whether or not it is possible to perform inference processing using the inference engine 73A. For example, if the inference engine 73A is currently processing another image file, or if the shutter switch 62 is being held down by the user and continuous shooting is being instructed, it is determined that it is not possible to perform inference processing using the inference engine 73A. In this case, the process proceeds to step S308, which will be described later. On the other hand, if none of the above conditions apply, it is determined that it is possible to perform inference processing using the inference engine 73A. In this case, the process proceeds to step S306.

[0047] In step S306, the system control unit 50A performs inference processing on the HEIF image file 200 using the inference engine 73A. The inference engine 73A takes the HEIF image file 200 as input and performs inference processing using the inference model recorded in the inference model recording unit 72A. In this inference processing, subject areas in the image data are identified from the image data and annotation information contained in the HEIF image file 200, and the classification result of the subject corresponding to each subject area is output as the inference result. In addition to the inference result, information related to the inference processing, such as operational debug information and logs during the inference processing, may also be output.

[0048] Next, in step S307, the system control unit 50A records the subject classification result obtained in step S306 as annotation information in the HEIF image file 200. The HEIF image file 200 only needs to contain the subject classification result output as the inference result of the inference model as annotation information. Therefore, the subject classification result can be recorded directly in the HEIF image file 200, and the recording format and style, such as text or binary, are not specified. By recording both the image data and the subject classification result in the HEIF image file 200 in this way, management becomes easier than managing these data individually. Furthermore, the system control unit 50A records the management version and debug information of the inference model held in the inference model recording unit 72A in the HEIF image file 200 as inference model management information.

[0049] Next, in step S308, the system control unit 50A determines whether or not it has received a user instruction to complete shooting. For example, the user can indicate completion of shooting by setting a mode other than shooting mode using the mode switching switch on the operation unit 63, or by setting the power to OFF using the power button. If it is determined in step S308 that the user has not received an instruction to complete shooting, the process returns to step S301. If it is determined in step S308 that the user has received an instruction to complete shooting, this process ends.

[0050] In this way, in this embodiment, an HEIF image file 200 is generated that includes image data generated by the digital camera 100 when it photographs a subject, and annotation information such as the classification results of the subject.

[0051] Furthermore, the digital camera 100 in this embodiment can also generate JPEG image files, and for example, it can convert the HEIF image file 200 described above into a JPEG image file.

[0052] Here, JPEG image files have a smaller storage capacity for annotation information compared to other image file formats such as HEIF. Therefore, when converting HEIF image files to JPEG image files, it is not possible to record all the annotation information recorded in the original image file into the JPEG image file. As a result, some of the annotation information recorded in the original image file cannot be used in the converted image file.

[0053] In contrast, in this embodiment, if the annotation information contained in the HEIF image file is a predetermined annotation information with a data volume exceeding a predetermined value, the annotation information is recorded in the application marker described later. If the annotation information contained in the HEIF image file is the predetermined annotation information described above, the link information of the annotation information is recorded in the application marker described later.

[0054] Figure 4 is a diagram illustrating the structure of a JPEG image file 400 used in this embodiment. As shown in Figure 4, the JPEG image file 400 has a marker SOI 401 at its beginning indicating the start of the image file, and an application marker (labeled "APP1" in Figure 4) 402 is recorded on the line following the marker SOI 401.

[0055] The application marker 402 is an area for recording annotation information, and its recording capacity is smaller than that of the 'meta'BOX 202 in the HEIF image file 200. The application marker 402 records the APP1 identification code 403. A typical APP1 identification code 403 is, for example, 'Exif'. Various metadata is recorded in the application marker 402. Specifically, the application marker 402 records the main image item definition 404, the related image item definition 405, the area item definition 406, the relationship between the main image and the related image 407, and the relationship between the main image and the attribute information of the main image 408. Furthermore, the application marker 402 records the relationship between the related image and the attribute information of the related image 409, the attribute information of the main image 410, the attribute information of the related image 411, and the attribute information of the area 412.

[0056] The item definition 404 of the main image records the same information as the item definition 204 of the main image described above. The item definition 405 of the related image records the same information as the item definition 205 of the related image described above. The item definition 406 of the region records the same information as the item definition 206 of the region described above. The relationship between the main image and the related image 407 records the same information as the relationship between the main image and the related image 208 described above. The relationship between the main image and the attribute information of the main image 408 records the same information as the relationship between the main image and the attribute information of the main image 209 described above. The relationship between the related image and the attribute information of the related image 409 records the same information as the relationship between the related image and the attribute information of the related image 210 described above. The attribute information of the main image 410 records the same information as the attribute information 212 of the main image described above. The attribute information of the related image 411 records the same information as the attribute information 213 of the related image described above. The attribute information of the region 412 records the same information as the attribute information 214 of the region described above.

[0057] The line following the region attribute information 412 records the application marker (labeled "APP2" in Figure 4) 413. Like the application marker 402, the application marker 413 is a region for recording annotation information, etc., and its recordable capacity is smaller than that of the 'meta'BOX 202 in the HEIF image file 200. The application marker 413 records the identification code 414 for APP2. The application marker 413 also records link information for related image data as metadata. Specifically, the application marker 413 records the location information 415 of the related image. The location information 415 of the related image records the size of the related image data 419 (described later) and the location of the SOI 418 marker indicating the start of the related image data 419.

[0058] The line following the location information 415 of the related image contains the main image data 416. The main image data 416 is the main image data of the JPEG image file 400. The line following the main image data 416 contains the marker EOI 417, which indicates the end of the main image data. The line following the marker EOI 417 contains the marker SOI 418, which indicates the start of the related image data, and the related image data 419, in that order. The line following the related image data 419 contains the marker EOI 420, which indicates the end of the related image data.

[0059] Figure 5A is a flowchart showing the procedure for the format conversion process performed in the digital camera 100 shown in Figure 1. The format conversion process in Figure 5A is realized by the system control unit 50A executing a program stored in the non-volatile memory 51 or the like. The format conversion process in Figure 5A is performed when the digital camera 100 receives an image conversion instruction from the user via the operation unit 63. The user selects at least one image file to be processed from among multiple image files stored on the external recording medium 91 and issues an image conversion instruction. As an example, Figure 5A describes the case where the user selects one HEIF image file and issues an image conversion instruction to convert this HEIF image file to a JPEG image file.

[0060] In Figure 5A, first, in step S500, the system control unit 50A performs metadata reading processing for the HEIF image file selected by the user. Specifically, the system control unit 50A reads the information recorded in 'ftype'BOX201 and 'meta'BOX202, respectively, of the HEIF image file into memory 25.

[0061] Next, in step S501, the system control unit 50A performs metadata analysis processing on the HEIF image file. In this metadata analysis processing, the system control unit 50A analyzes the information from 'ftype'BOX201 and 'meta'BOX202 that were read into memory 25 in step S500, and extracts the metadata recorded in the HEIF image file. Through this metadata analysis processing, the location information 216 of the main image and the location information 217 of related images recorded in the HEIF image file are obtained.

[0062] Next, in step S502, the system control unit 50A performs a reading process of the main image data recorded in the HEIF image file. Specifically, the system control unit 50A uses the position information 216 of the main image obtained by the metadata analysis process to read the main image data 219 from the HEIF image file and writes the main image data 219 to the memory 25.

[0063] Next, in step S503, the system control unit 50A performs format conversion processing on the main image data 219. Specifically, the system control unit 50A decodes the main image data 219 that was written to the memory 25 in step S502. Furthermore, the system control unit 50A uses a conversion engine (not shown) to encode the decoded main image data 219 as main image data 416 to be recorded in a JPEG image file.

[0064] Next, in step S504, the system control unit 50A reads the related image data recorded in the HEIF image file. Specifically, the system control unit 50A reads the related image data 220 from the HEIF image file using the position information 217 of the related image obtained by the metadata analysis process, and writes the related image data 220 to the memory 25.

[0065] Next, in step S505, the system control unit 50A performs a format conversion process on the associated image data 220. Specifically, the system control unit 50A decodes the associated image data 220 that was written to the memory 25 in step S504. Furthermore, the system control unit 50A uses a conversion engine (not shown) to encode the decoded associated image data 220 as associated image data 419 to be recorded in a JPEG image file.

[0066] Next, in step S506, the system control unit 50A performs the header creation process shown in Figure 5B, which will be described later, to create a header for the JPEG image file that will become the converted image file. Then, in step S507, the system control unit 50A performs the JPEG image file saving process shown in Figure 5C, which will be described later, to save the JPEG image file converted from the HEIF image file to the external recording medium 91, and terminate this process.

[0067] Figure 5B is a flowchart showing the procedure for the header creation process in step S506 of Figure 5A.

[0068] In Figure 5B, in step S520, the system control unit 50A performs a first memory area allocation process. This first memory area allocation process allocates a first memory area in memory 25 that will be used to create the header for the JPEG image file, which will be the converted image file.

[0069] Next, in step S521, the system control unit 50A performs metadata acquisition processing for the HEIF image file. Specifically, the system control unit 50A reads the information from 'ftype'BOX201 and 'meta'BOX202, which were written to memory 25 in step S500, from memory 25.

[0070] Next, in step S522, the system control unit 50A performs a first application marker creation process. The first application marker creation process creates an application marker 402 in the JPEG image file. Here, as an example of an application marker 402, APP1 is created. Note that the application marker 402 created in step S522 is not limited to APP1; another application marker corresponding to the identification code created in step S523, which will be described later, may also be created.

[0071] Next, in step S523, the system control unit 50A performs a first identification code creation process. This first identification code creation process creates an identification code 403 for APP1 in the JPEG image file. Here, as an example of an identification code 403 for APP1, 'Exif' is created. Note that the identification code 403 for APP1 created in step S523 is not limited to 'Exif', and other identification codes may be created.

[0072] Next, in step S524, the system control unit 50A determines whether the metadata acquired in step S521 includes annotation information. If it is determined in step S524 that the metadata acquired in step S521 includes annotation information, the process proceeds to step S525. If it is determined in step S524 that the metadata acquired in step S521 does not include annotation information, the process proceeds to step S535, which will be described later.

[0073] In step S525, the system control unit 50A starts an annotation information generation process to generate annotation information to be recorded in the JPEG image file. The annotation information generation process performs the operations in steps S526 to S533. The annotation information generation process is performed for all annotation information to be recorded in the HEIF image file.

[0074] In step S526, the system control unit 50A obtains annotation information from the metadata acquired in step S521. Next, the system control unit 50A determines whether the acquired annotation information is a predetermined annotation information with a data volume exceeding a predetermined value. Specifically, in step S527, the system control unit 50A determines whether the acquired annotation information is related image data. In step S527, for example, the system control unit 50A determines whether the acquired annotation information is related image data based on the information in the 'item definition' BOX 203 of the metadata acquired in step S521.

[0075] If, in step S527, it is determined that the acquired annotation information is not related image data, the process proceeds to step S528. In step S528, the system control unit 50A performs a first generation process. In the first generation process, the acquired annotation information, that is, annotation information other than related image data, is generated as annotation information to be recorded in the JPEG image file 400 in its original state. For example, if the acquired annotation information is region attribute information 214, the system control unit 50A saves the region attribute information 214 in its original state to the first storage area. This annotation information is recorded as region attribute information 412 in the application marker 402 of the JPEG image file 400 by the JPEG image file saving process shown in Figure 5C, which will be described later. In this embodiment, annotation information in the HEIF image file other than predetermined annotation information that has a data amount exceeding a predetermined value is recorded in its original state in the application marker of the JPEG image file. Next, the process proceeds to step S534, which will be described later.

[0076] In step S527, if the acquired annotation information is determined to be related image data, the process proceeds to step S529, which will be described later. In step S529, the system control unit 50A performs a second generation process. In the second generation process, annotation information that is recorded in the application marker 402 of the JPEG image file and is related to the related image data is generated. For example, annotation information corresponding to the relationship 409 between the related image and the attribute information of the related image, and annotation information corresponding to the attribute information 411 of the related image are generated. The generated annotation information is stored in the first memory area secured in step S520.

[0077] Next, in step S530, the system control unit 50A performs a second memory area allocation process. The second memory area allocation process allocates a second memory area in memory 25 that is different from the first memory area. The second memory area is a memory area used to create annotation information recorded in the application marker 413 of a JPEG image file.

[0078] Next, in step S531, the system control unit 50A performs a second application marker creation process. This second application marker creation process creates an application marker 413 in the JPEG image file. Here, as an example of an application marker 413, APP2 is created. Note that the application marker 413 created in step S531 is not limited to APP2; another application marker corresponding to the identification code created in step S532, which will be described later, may also be created.

[0079] Next, in step S532, the system control unit 50A performs a second identification code creation process. The second identification code creation process creates the identification code 414 for APP2 in the JPEG image file.

[0080] Next, in step S533, the system control unit 50A performs a third generation process. In the third generation process, link information of the associated image data to be recorded in the application marker 413 of the JPEG image file is generated. For example, link information corresponding to the location information 415 of the associated image is generated. The generated link information is stored in the second memory area secured in step S530. The various information generated in steps S529 and S533 is recorded in the application markers 402 and 413 of the JPEG image file by the JPEG image file saving process shown in Figure 5C, which will be described later. Thus, in this embodiment, among the annotation information in the HEIF image file, predetermined annotation information that exceeds a predetermined data amount is not recorded in the application marker of the JPEG image file, and its link information is recorded instead. Note that if the location information of the associated image data is determined in the JPEG image file saving process shown in Figure 5C, which will be described later, the link information corresponding to the location information 415 of the associated image may be updated at the time the location information is determined. After that, the process proceeds to step S534.

[0081] Once processing of all annotation information recorded in the HEIF image file is complete, the system control unit 50A terminates the annotation information generation process in step S534.

[0082] Next, in step S535, the system control unit 50A generates metadata other than annotation information. For example, if the identification code 403 of APP1 is 'Exif', then in step S535, metadata such as the date and time of shooting and shooting conditions are generated. After that, this process ends.

[0083] Figure 5C is a flowchart showing the procedure for saving a JPEG image file in step S507 of Figure 5A.

[0084] In Figure 5C, in step S540, the system control unit 50A writes information related to the header of the JPEG image file generated by the header creation process described above, for example, information stored in the first recording area, to the external recording medium 91. Next, in step S541, the system control unit 50A writes information recorded in the application marker 413 (APP2) generated by the header creation process described above, for example, information stored in the second recording area, to the external recording medium 91. Next, in step S542, the system control unit 50A writes the main image data 416 of the JPEG image file 400 to the external recording medium 91. Next, in step S543, the system control unit 50A writes the associated image data 419 of the JPEG image file 400 to the external recording medium 91. Next, in step S544, the system control unit 50A writes other data to the external recording medium 91. After that, the process ends.

[0085] According to the embodiment described above, if the annotation information contained in the HEIF image file is not the predetermined annotation information that would result in a data volume exceeding a predetermined value, this annotation information is recorded in the application marker of the JPEG image file. If the annotation information contained in the HEIF image file is the predetermined annotation information, the link information of this annotation information is recorded in the application marker of the JPEG image file. This makes it possible to provide a converted image file that can utilize all the annotation information recorded in the image file before conversion.

[0086] Furthermore, in the above-described embodiment, since the predetermined annotation information is related image data, it is possible to provide a converted image file that can utilize related image data with a relatively large amount of data.

[0087] Furthermore, in the embodiment described above, the associated image data 419 is recorded in a region of the JPEG image file 400 that is different from the application markers 402 and 413. This allows the associated image data 419 to be read from the converted image file.

[0088] The present invention has been described above using the embodiments described above, but the present invention is not limited to the embodiments described above. For example, as shown in Figure 6, the associated image data 620 may be recorded in a different image file 618 from the converted JPEG image file 600. In this case, as link information for the associated image data 620, for example, the file name 615 of the associated image is recorded in the application marker 613 of the converted JPEG image file 600. The file name 615 of the associated image is the file name of the image file 618 in which the associated image data 620 is recorded. The configuration of the JPEG image file 600 differs from the JPEG image file 400 in Figure 4 described above in that the file name 615 of the associated image is recorded instead of the location information 415 of the associated image, and the associated image data 419 is not recorded. Otherwise, the configuration is basically the same as the JPEG image file 400 in Figure 4 described above.

[0089] Figure 7 is a flowchart showing an alternative procedure for the header creation process shown in Figure 5B. Note that the header creation process in Figure 7 is similar to that in Figure 5B; however, the differences between the two processes will be explained below.

[0090] In Figure 7, steps S701 to S707 are performed, which are the same processes as steps S520 to S527 described above.

[0091] If, in step S707, it is determined that the acquired annotation information is not related to the image data, the process proceeds to step S708, which is the same process as step S528 described above. Subsequently, steps S714 and S715, which are the same processes as steps S534 and S535 described above, are performed, and this process is terminated.

[0092] In step S707, if the acquired annotation information is determined to be related image data, the process proceeds to step S709, which is the same process as step S529 described above. Subsequently, steps S710 to S712 are performed, which are the same processes as steps S530 to S532 described above. Next, in step S713, the system control unit 50A performs a fourth generation process. In the fourth generation process, unlike the third generation process described above, the link information for the related image data is not the link information corresponding to the location information 415 of the related image, but the link information corresponding to the file name 615 of the related image. In this embodiment, the link information generated in step S713 is not limited to the file name of the image file on which the related image data is recorded, as long as it is information that can identify the related image data. For example, it may be an ID for identifying the related image data. The generated link information is stored in the second memory area secured in step S710. Subsequently, steps S714 and S715 are performed, which are the same processes as steps S534 and S535 described above, and this process ends.

[0093] Figure 8 is a flowchart showing an alternative procedure for saving a JPEG image file as shown in Figure 5C. Note that the JPEG image file saving process in Figure 8 is similar to that in Figure 5C, and the following explanation will focus on the differences between the two processes.

[0094] In Figure 8, steps S800 to S802 are performed, which are the same processes as steps S540 to S542 described above. Next, step S803 is performed, which is the same process as step S544 described above. Then, in step S804, the system control unit 50A writes the associated image data 620 to the external recording medium 91 as a separate image file 618, different from the JPEG image file 600. After that, this process is terminated.

[0095] Even in this configuration where related image data is recorded in a separate image file different from the converted image file, the related image data recorded in the original image file can still be used in the converted image file.

[0096] In the embodiments described above, the case where the predetermined annotation information exceeds a predetermined value is described as related image data. However, the predetermined annotation information is not limited to related image data. For example, the predetermined annotation information may be video data or audio data, which are not permitted to be recorded in image files under the JPEG standard.

[0097] If the specified annotation information is video data or audio data, the specified annotation information is recorded in a different image file, not in the converted image file. Furthermore, the application marker in the converted image file records link information for the specified annotation information. This link information includes the filename of the image file containing the specified annotation information, or an ID to identify the specified annotation information. In this way, even if the specified annotation information is video data or audio data, the same effects as the above-described embodiment can be achieved.

[0098] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of the system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0099] Furthermore, the disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An image processing apparatus comprising a conversion means for converting an image file of a first format, which includes image data and annotation information of said image data, into an image file of a second format, wherein the recordable capacity of a predetermined area for recording annotation information is smaller than that of the first format image file, wherein the conversion means records the annotation information in the predetermined area of ​​the second format image file if the annotation information included in the first format image file is not predetermined annotation information that results in a data amount of more than a predetermined value, and records the link information of the annotation information in the predetermined area of ​​the second format image file if the annotation information included in the first format image file is predetermined annotation information. (Configuration 2) The image processing apparatus according to Configuration 1, characterized in that the predetermined annotation information is related image data of the image data. (Configuration 3) The image processing apparatus according to Configuration 2, characterized in that the conversion means records the predetermined annotation information in an area of ​​the second format image file that is different from the predetermined area. (Configuration 4) The image processing apparatus according to Configuration 2, characterized in that the conversion means records the predetermined annotation information in a separate image file different from the image file of the second format. (Configuration 5) The image processing apparatus according to Configuration 1, characterized in that the predetermined annotation information is video data or audio data. (Configuration 6) The image processing apparatus according to Configuration 5, characterized in that the conversion means records the predetermined annotation information in a separate image file different from the image file of the second format. [Explanation of Symbols]

[0100] 50A System Control Unit 100 Digital Cameras 400 JPEG image files 402 Application Marker 413 Application Markers 415 Related image location information 600 JPEG image files 602 Application Marker 613 Application Markers 615 Related image filenames 618 image files 620 related image data

Claims

1. The system includes a conversion means for converting an image file of a first format, which includes image data and annotation information for said image data, into an image file of a second format, in which the recordable capacity of a predetermined area for recording annotation information is smaller than that of the image file of the first format. The conversion means is characterized in that, if the annotation information contained in the first format image file is not predetermined annotation information that results in a data amount of more than a predetermined value, it records the annotation information in the predetermined area of ​​the second format image file, and if the annotation information contained in the first format image file is predetermined annotation information, it records the link information of the annotation information in the predetermined area of ​​the second format image file.

2. The image processing apparatus according to claim 1, characterized in that the predetermined annotation information is related image data of the image data.

3. The image processing apparatus according to claim 2, characterized in that the conversion means records the predetermined annotation information in an area of ​​the second type of image file that is different from the predetermined area.

4. The image processing apparatus according to claim 2, characterized in that the conversion means records the predetermined annotation information in a separate image file different from the image file of the second format.

5. The image processing apparatus according to claim 1, characterized in that the predetermined annotation information is video data or audio data.

6. The image processing apparatus according to claim 5, characterized in that the conversion means records the predetermined annotation information in a separate image file different from the image file of the second format.

7. The method includes a conversion step of converting an image file of a first format, which includes image data and annotation information of said image data, into an image file of a second format, in which the recordable capacity of a predetermined area for recording annotation information is smaller than that of the image file of the first format. The control method for an image processing apparatus, characterized in that the conversion step includes recording the annotation information in a predetermined area of ​​the second image file if the annotation information contained in the first image file is not predetermined annotation information with a data amount equal to or greater than a predetermined value, and recording the link information of the annotation information in a predetermined area of ​​the second image file if the annotation information contained in the first image file is predetermined annotation information.

8. A program that causes a computer to execute a control method for an image processing device, The control method for the image processing device is: The method includes a conversion step of converting an image file of a first format, which includes image data and annotation information of said image data, into an image file of a second format, in which the recordable capacity of a predetermined area for recording annotation information is smaller than that of the image file of the first format. The program is characterized in that, if the annotation information contained in the first format image file is not predetermined annotation information that results in a data amount of more than a predetermined value, it records the annotation information in the predetermined area of ​​the second format image file, and if the annotation information contained in the first format image file is predetermined annotation information, it records the link information of the annotation information in the predetermined area of ​​the second format image file.

Images