Information processing device, control method for image processing device, and program

Abstract

The present invention is characterized by including an acquisition means for acquiring an image file that includes a plurality of pieces of annotation information, and a control means for performing control so as to generate and record a new image file on the basis of the image file acquired by the acquisition means, the control means performing control of: recording, among the plurality of pieces of annotation information included in the image file, annotation information added to the image file by an imaging device in a first recording area of the new image file; and recording annotation information added to the image file by a device different from the imaging device in a second recording area of the new image file, the second recording area having a lower reproduction compatibility than the first recording area.

Description

Information processing device, control method for image processing device, and program

[0001] The present invention relates to an image processing device that processes image files of images captured by a camera or the like.

[0002] In recent years, AI technologies such as deep learning have been increasingly utilized in various technical fields.

[0003] For example, conventionally, digital cameras (hereinafter referred to as cameras) and the like have been equipped with a function to detect human faces from captured images, but Patent Document 1 and others propose technology that does not limit the detection target to humans, but can accurately detect and recognize animals such as dogs and cats.

[0004] A technology has been proposed for recording information about a detected object as annotation information in association with image data, and Patent Document 2 proposes a method for recording annotation information when converting images between different image formats. Compared to image files in HEIF (High Efficiency Image File Format), image files in JPEG format have a smaller recording capacity for recording annotation information. Therefore, when converting from HEIF to JPEG, link information for annotation information that cannot be recorded in JPEG due to capacity limitations is recorded in a recordable area. This technology then discloses that by recording annotation information at the link destination, it is possible to record a large amount of annotation information even in JPEG.

[0005] JP 2015-99559 A JP 2023-173960 A

[0006] The prior art disclosed in the aforementioned patent document allows for recording of a large amount of annotation information by converting from HEIF to JPEG, recording annotation information that cannot be recorded as a separate file, and recording link information of the separate file in the JPEG file. However, when playing back a JPEG file on a playback device, if the link information in the annotation information is invalid, if the link destination cannot be connected, or if the data at the link destination has been deleted, the annotation information cannot be acquired. Furthermore, even if the annotation information at the link destination can be acquired, if the annotation information was recorded on a device other than the imaging device, playback compatibility may be low and playback may not be possible.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device that records annotation information in an image file in such a way that the annotation information can be reproduced on a reproduction device different from the device that recorded the annotation information.

[0008] In order to achieve the above object, the image processing device of the present invention has an acquisition means for acquiring an image file containing multiple pieces of annotation information, and a control means for controlling the generation and recording of a new image file based on the image file acquired by the acquisition means, and is characterized in that the control means controls the recording of annotation information added to the image file by an imaging device, out of the multiple pieces of annotation information included in the image file, in a first recording area of ​​the new image file, and the recording of annotation information added to the image file by a device other than the imaging device, in a second recording area of ​​the new image file, which is a second recording area having lower playback compatibility than the first recording area.

[0009] According to the present invention, when annotation information is recorded in an image file, the annotation information can be recorded so as to be reproducible on a reproducing device different from the device on which the annotation information was recorded.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a camera according to an embodiment. FIG. 2 is a block diagram showing an example of the configuration of a PC according to an embodiment. FIG. 3 is a block diagram showing processing units of the camera and PC according to an embodiment. FIG. 4A is a diagram showing an example of the configuration of a HEIF image file according to the first embodiment. FIG. 4B is a diagram showing an example of metadata. FIG. 5A is a diagram showing an example of a display of an image file according to the first embodiment. FIG. 5B is a diagram showing an example of a display of an image file according to the first embodiment. FIG. 6 is a flowchart of the operation of a PC according to the first embodiment. FIG. 7 is a detailed flowchart of the operation of a PC according to the first embodiment. FIG. 8A is a diagram showing an example of the configuration of a JPEG image file according to the first embodiment. FIG. 8B is a diagram for explaining a display example. FIG. 9 is a diagram for explaining the configuration of a HEIF image file according to the first embodiment. FIG. 10A is a diagram for explaining the configuration of a JPEG image file and another file according to the first embodiment. FIG. 10B is a diagram for explaining the configuration of a JPEG image file and another file according to the first embodiment.

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] <System Configuration> Note that, in the following, a camera (image capture device) and a PC (information processing device) are exemplified as image processing devices, but the image processing device of the present invention is not limited to these. The image processing device of the present invention may be any device that reads images recorded in a recording device (recording medium) and displays and outputs them on a display device or outputs them as an image file, and may be, for example, a smartphone, a tablet PC, or the like.

[0013] FIG. 1 is a block diagram showing an example of the configuration of a camera 100 according to an embodiment of the present invention.

[0014] The barrier 10 is a protective member that covers the imaging section of the camera 100, including the photographing lens 11, to prevent the imaging section from becoming dirty or damaged. The photographing lens 11 focuses an optical image on the imaging surface of the imaging element 13. The shutter 12 has an aperture function. The imaging element 13 converts the optical image focused on the imaging surface into an electrical signal. The imaging element 13 is composed of, for example, a CCD or CMOS element. The A / D converter 15 converts the analog signal output from the imaging element 13 into a digital signal. The digital signal converted by the A / D converter 15 is written to memory 25 as a so-called R digital data. At the same time, development parameters corresponding to each piece of RAW image data are generated based on information at the time of shooting and written to memory 25. The development parameters include various parameters used in image processing for recording in JPEG format, such as exposure settings, white balance, color space, and contrast.

[0015] The timing generator 14 supplies clock signals and control signals to the image sensor 13, the A / D converter 15, and the D / A converter 21. The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50. The image processing unit 20 performs various image processing such as predetermined pixel interpolation, color conversion, correction, and resizing on data from the A / D converter 15 or data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and arithmetic processing using captured image data and provides the obtained arithmetic results to the system control unit 50. The system control unit 50 controls the exposure control unit 40 and the distance measurement control unit 41 based on the provided arithmetic results, thereby realizing AF (autofocus) processing, AE (autoexposure) processing, and EF (pre-flash) processing.

[0016] The image processing unit 20 performs predetermined calculations using the captured image data and also performs AWB (auto white balance) processing based on the calculation results. Furthermore, the image processing unit 20 reads images stored in the memory 25 and performs compression or decompression processing such as JPEG, MPEG-4 AVC, or HEVC (High Efficiency Video Coding), or lossless compression of uncompressed RAW data. The image processing unit 20 writes the processed data to the memory 25.

[0017] The image processing unit 20 also performs predetermined arithmetic processing using captured image data to edit various types of image data. Specifically, it can perform a trimming process to adjust the display range and size of the image by hiding unnecessary parts around the image data, and a resizing process to change the size of the image data or screen display elements by enlarging or reducing them. Furthermore, it performs RAW development, which performs image processing such as color conversion on data that has been compressed or decompressed using lossless compression or decompression, and converts it to JPEG format to create image data. It can also perform video clipping, which clips out specified frames from video formats such as MPEG-4, converts them to JPEG format, and saves them.

[0018] The image processing unit 20 also performs processing such as superimposing an OSD (On-Screen Display) such as a menu or arbitrary characters to be displayed on a display unit 23 (to be described later) together with the display image data.

[0019] Furthermore, the image processing circuit 20 performs subject detection processing to detect the subject present in the image data and the subject distance information obtained from the image sensor 13 at the time of shooting, etc. Detectable information includes the position and size of the subject within the image, as well as the inclination and likelihood.

[0020] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The raw image data generated through the A / D converter 15 is written into the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22.

[0021] The image display memory 24, D / A converter 21, and TFT LCD constitute the display unit 23. The image data for display written in the image display memory 24 is displayed on the display unit 23 via the D / A converter 21. If the captured image data is displayed sequentially using the display unit 23, an electronic viewfinder function for displaying live images can be realized. The memory 25 stores captured still images and moving images. The memory 25 has a storage capacity sufficient to store a predetermined number of still images and moving images for a predetermined period of time. The memory 25 can also be used as a working area for the system control unit 50A.

[0022] The exposure control unit 40 controls the shutter 12, which has an aperture function. The exposure control unit 40 also has a flash dimming function by working in conjunction with a flash 44. The distance measurement control unit 41 controls focusing of the photographic lens 11. The zoom control unit 42 controls zooming of the photographic lens 11. The barrier control unit 43 controls the operation of the barrier 10, which is a protective member. The flash 44 has an AF assist light projection function and a flash dimming function.

[0023] The system control unit 50A controls the entire camera 100. The nonvolatile memory 51 is an electrically erasable and recordable nonvolatile memory. For example, an EEPROM or the like is used as the nonvolatile memory 51. The nonvolatile memory 51 stores not only programs but also map information and the like. The system control unit 50A also functions as a processing unit that executes processing based on the programs stored in the nonvolatile memory 51. The processing unit will be described in detail later with reference to FIG. 3.

[0024] The shutter switch 61 (SW1) is turned ON during operation of the shutter button 60 to instruct the start of operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing. The shutter switch 62 (SW2) is turned ON when the shutter button 60 is completely operated to instruct the start of a series of photographing operations including exposure processing, development processing, and recording processing. In the exposure processing, a signal read from the image sensor 13 is written as raw data to the memory 25 via the A / D converter 15 and the memory control unit 22. In the development processing, the raw data written to the memory 25 is developed using calculations in the image processing unit 20 and the memory control unit 22, and then written to the memory 25 as image data. In the recording processing, the image data is read from the memory 25, compressed by the image processing unit 20, stored in the memory 25, and then written to the external recording medium 91 via the card controller 90.

[0025] The operation unit 63 includes various buttons, a touch panel, etc. For example, the operation unit 63 includes a power button, a menu button, a mode switch for switching between shooting mode, playback mode, and other special shooting modes, a cross key, a set button, a macro button, and a multi-screen playback page break button. Furthermore, for example, the operation unit 63 includes a flash setting button, a single-shot / continuous-shot / self-timer switching button, a menu movement + (plus) button, a menu movement - (minus) button, a shooting quality selection button, an exposure compensation button, a date / time setting button, etc.

[0026] When recording image data as an image file on the external recording medium 91, the metadata generation and analysis unit 70 generates various metadata, such as Exif (Exchangeable image file format) standard, for the image data based on information at the time of shooting. Furthermore, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata attached to the image data. Examples of metadata include shooting setting information at the time of shooting, image data information related to the image data, subject recognition information (depth information, subject characteristics, etc.), and annotation information. Furthermore, when recording moving image data, the metadata generation and analysis unit 70 can also generate and attach metadata for each frame. In the camera 100 of this embodiment, when recording an image file through a shooting operation, the system control unit 50A generates the above-mentioned metadata using the metadata generation and analysis unit 70, and the system control unit 50A stores the generated metadata in the metadata recording area of ​​the image file. The system control unit 50A then controls the card controller 90 to record the image file thus generated on the external recording medium 91.

[0027] The power supply 80 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The power supply control unit 81 supplies power from the power supply 80 to each unit of the camera 100. The card controller 90 transmits and receives data to and from an external recording medium 91 such as a memory card. The external recording medium 91 may be, for example, a memory card, and records images (still images and videos) captured by the camera 100.

[0028] The inference engine 73A uses an inference model stored in the inference model recording unit 72A to perform inference on image data input from the system control unit 50A. The inference model may be input from an external device such as the external device 101 via the communication unit 71A and recorded in the inference model recording unit 72A, or may be an inference model that has been retrained by the learning unit 74A described below. The inference engine 73A has a neural network design 73a.

[0029] In the neural network design 73a, an intermediate layer (neurons) is arranged between the input layer and the output layer. Image data is input to the input layer from the system control unit 50A. Several layers of neurons are arranged in the intermediate layer. The number of neuron layers is determined appropriately in the design, and the number of neurons in each layer is also determined appropriately in the design. The intermediate layer is weighted based on the inference model recorded by the inference model recording unit 72A. Annotation information corresponding to the image input to the input layer is output to the output layer.

[0030] The inference engine 73A can be updated from an external device 101 or the like, and can handle various inference models.

[0031] The learning unit 74A will now be described. The learning unit 74A receives a request from the system control unit 50A or the like and performs re-learning of the inference model. The learning unit 74A has a teacher data recording unit 74a. The teacher data recording unit 74a records information about teacher data for the inference engine 73A. The learning unit 74A can re-learn the inference engine 73A using the teacher data recorded in the teacher data recording unit 74a and update the inference engine 73A using the inference model recording unit 72A.

[0032] When an inference model is updated externally or through re-learning by the learning unit 74A, a management version is maintained in the inference model recording unit 72A or the like so that each inference model can be identified.

[0033] Furthermore, in this embodiment, the inference engine 73A performs inference using a neural network, but any method may be used as long as it is capable of classifying, inferring, and analyzing subjects included in an image.

[0034] The communication unit 71A has a communication circuit for transmitting and receiving. Specifically, the communication path may be wireless communication such as Wi-Fi or Bluetooth, or wired communication such as Ethernet or USB. The communication unit 71A can communicate with the communication unit 71B of the external device 101. The communication unit 71A functions as a communication unit that transmits and receives not only image files with annotation information added thereto created by the inference engine 73A, but also various information such as inference models and training data to and from the system control unit 50A and the system control unit 50B. The information to be transmitted can be restricted depending on whether the external device 101 is associated with a camera.

[0035] The external device 101 has a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system recording unit 50B, and a communication unit 71B. Of course, devices that do not have these components are also included. The learning unit 74B creates an inference model upon request from an external device, such as the inference engine 73B or the system recording unit 50B. The inference model recording unit 72B records the inference model sent from the camera 100 and the inference model created by the learning unit 74B.

[0036] FIG. 2 is a block diagram showing an example of the configuration of the PC 200 according to the embodiment of the present invention.

[0037] The control unit 201 is, for example, a central processing unit (CPU) and controls the entire PC 200. The control unit 201 also functions as a processing unit that executes processing based on a program stored in a ROM 202. The processing unit will be described in detail later with reference to FIG. 3.

[0038] A read only memory (ROM) 202 non-temporarily stores programs and parameters.

[0039] A random access memory (RAM) 203 temporarily stores programs and data supplied from external devices.

[0040] The recording medium 204 is a hard disk or flash memory fixedly installed in the PC 200, or an optical disk, magnetic card, optical card, IC card, memory card, or the like that is detachable from the PC 200.

[0041] The operation unit 205 accepts user operations on the PC 200. It also has a function of selecting, in response to an editor's operation, one of still images that satisfy predetermined conditions and are displayed on the display unit 206. The operation member used by the user to perform the operation here may be a button or a touch panel provided on the PC 200, or a keyboard or a mouse that is detachable from the PC 200.

[0042] The display unit 206 displays data stored in the PC 200, data supplied from an external device, etc. It also has a function of displaying metadata added to images recorded on the recording medium 204.

[0043] The communication unit 207 communicates with external devices such as the camera 100 .

[0044] A system bus 208 communicatively connects the components of the PC 200 .

[0045] PC 200 can communicate with communication unit 71A of camera 100 via communication unit 207 to acquire image files from camera 100 and display the acquired image files on display unit 206. In addition, control unit 201 can generate new image files from the acquired image files and record them on recording medium 204. Image files may be acquired not via communication unit 207 but via a recording medium by attaching recording medium 91, on which image files captured by camera 100 are recorded, to PC 200 as recording medium 204.

[0046] FIG. 3 is a block diagram showing the configuration of the processing units included in the control units of the camera 100 and the PC 200. As shown in FIG.

[0047] The system control unit 50A of the camera 100 and the control unit 201 of the PC 200 also function as the processing unit in Fig. 3 by loading predetermined programs stored in the non-volatile memory 51 and the ROM 202 into the working RAM and executing them. The following describes each unit (300 to 306) included in the processing unit (system control unit 50A or control unit 201).

[0048] The image reading unit 300 reads and acquires an image file from a storage medium. The image file may be acquired via the communication unit 207.

[0049] The annotation information acquisition unit 301 acquires annotation information recorded in the image file from the image file read by the image reading unit 300 .

[0050] The image storage unit 302 executes a process of storing (recording) the image file generated by the processing unit in a storage medium.

[0051] The annotation information adding unit 303 adds annotation information to an image file that is newly saved in a storage medium by the image saving unit 302. In this embodiment, the annotation information is recorded in the metadata area of ​​the image file when it is added, but it may also be recorded as a separate file related to the image file.

[0052] The image display control unit 304 performs processing to display, on the display unit, the image data of the image file read by the image reading unit 300 or the image file generated by the processing unit.

[0053] The annotation information display control unit 305 performs processing to display on the display unit the annotation information acquired by the annotation information acquisition unit 301 and the annotation information added to the image file by the annotation information addition unit 303. In cooperation with the image display control unit 304, the annotation information and the image data of the image file may be displayed superimposed on each other on the display unit, or the image data and the annotation information may be displayed separately.

[0054] The annotation information generation unit 306 detects a subject and generates annotation information using the inference engine 73A or 73B in the camera 100, or using a predetermined inference program in the PC 200. Since the camera 100 has the metadata generation and analysis unit 70, the metadata may be generated by the metadata generation and analysis unit 70 without providing the annotation information generation unit 306.

[0055] In this embodiment, when capturing an image, the camera 100 generates an image file 400 in HEIF format as an image file and records it on the external recording medium 91. After capturing an image, the camera 100 or the PC 200 can generate metadata using the image data of the image file 400 by the annotation information generation unit 306, and can add annotation information to the image file 400.

[0056] An image file in HEIF format and metadata recorded in the image file will be described with reference to FIGS. 4A and 4B.

[0057] FIG. 4A is a diagram showing the structure of a HEIF image file 400 used in this embodiment.

[0058] As shown in FIG. 4A , a HEIF image file 400 is composed of multiple areas called BOXes, and data is recorded in a tree structure. Data may be recorded directly in a BOX, or another BOX may be recorded as a child element. The HEIF image file 400 includes an 'ftyp' BOX 405 that records the image file type, a 'meta' BOX 406 that records metadata including annotation information, and an 'mdat' BOX 408 that records image data. Note that FIG. 4A shows the minimum number of BOXes required for explaining the present invention, but the HEIF image file 400 may include BOXes other than those described above.

[0059] The 'meta' BOX 406 has, as child elements, a 'camera parameters' BOX 407 and multiple annotation BOXes. Fig. 4A shows an example in which four annotation BOXes ('annotation 1' BOX 401, 'annotation 2' BOX 402, 'annotation 3' BOX 403, and 'annotation 4' BOX 404) are recorded.

[0060] The 'camera parameters' box 407 stores, as child elements, the parameters of the camera 100 that were recorded when an image was captured. Fig. 4A shows an example in which shooting mode information 4071, AF frame information 4072, and subject blur level information 4073 are stored in the 'camera parameters' box 407. If there are any camera parameters other than these, they are recorded in the 'camera parameters' box 407. The other camera parameters include shooting setting information such as exposure settings at the time of shooting, camera model information, camera manufacturer information, etc.

[0061] The shooting mode information 4071 stores the shooting mode set by the user when capturing the HEIF image file 400. Examples of shooting modes include portrait mode, landscape mode, and sports mode.

[0062] The AF frame information 4072 stores the AF frame position set by the user or the camera 100 when capturing the HEIF image file 400, and multiple AF frames may be set. The AF frame position is defined by a position on the image data, and coordinates relative to the image data, the width and height of the rectangle, etc. are recorded. For example, by recording the AF frame information 4072, an AF frame can be displayed on the image, like the AF frame 451 shown in FIG. 5A.

[0063] The subject blur level information 4073 stores the degree of blur for the subject calculated from information at the time of shooting of the image data 409 in the HEIF image file 400 and coordinate information on the image data, and can store the degrees of blur for multiple subjects in the image. Note that, in this embodiment, an example will be described in which the degrees of blur are described in a form corresponding to the area of ​​each annotation, but this is not limiting.

[0064] Information 4011 about annotation A is recorded as a child element in the 'annotation 1' box 401, and as shown in FIG. 4A, the area of ​​annotation A and a description of annotation A are recorded. The annotation area information includes the shape and coordinates of the area of ​​the subject indicated by the annotation information in the image data 409. In this embodiment, the annotation area is recorded using the coordinates and size (width, height) in the image data, but area information may be recorded using other methods. The annotation description includes the name and attributes of the subject included in the annotation area as character string information.

[0065] Similarly, annotation information B, C, and D are recorded in 'annotation 2' BOX 402, 'annotation 3' BOX 403, and 'annotation 4' BOX 404, respectively. Although not described in the embodiments of the present invention, if there are five or more annotations, they are also recorded in 'meta' BOX 406 in the same way.

[0066] Image data 409 is recorded in an 'mdat' BOX 408 for recording image data.

[0067] The definition of the child elements of each BOX is not limited to the structure shown in Fig. 4A, and may be described, for example, in a BOX containing only the annotation area, or in a BOX containing only the annotation explanation, etc. Furthermore, multiple pieces of annotation information may be recorded in one annotation BOX. When recording multiple pieces of annotation information in one annotation BOX, child elements may be further provided within the annotation BOX, and the pieces of annotation information may be recorded in a tree structure (hierarchical structure).

[0068] FIG. 4B shows an example of metadata 406 recorded in the image file 400 .

[0069] The shooting mode information 4071, AF frame information 4072, and blur degree information 4073 of the camera parameter box 407 store information generated by the metadata generation and analysis unit 70 of the camera 100 at the time of shooting. The AF frame information stores information 451 about the position and size of the AF frame at the time of shooting. Information about the type of subject may also be stored. The subject blur degree information 4073 stores identification information of the annotation box corresponding to the blur degree information. By referring to this identification information, it is possible to associate the annotation information with the blur degree information.

[0070] In FIG. 4B , annotation 1 BOX 401 stores annotation information 452-455 that was added to image file 400 when image data 409 was captured by camera 100. Annotation 2 BOX 402 stores annotation information 461-465 that was generated by annotation information generation unit 306 on PC 200 or the like after capture and added to the image file by annotation addition unit 303. As described above, the annotation BOX stores the area and description for each piece of annotation information (452-455, 461-465). Annotation 2 BOX 402 records not only the area and description for annotation information 461-465, but also information on the degree of blur. The degree of blur corresponding to annotation information 461-465 was not recorded by camera 100 during capture. Therefore, it cannot be recorded in blur degree information 4073 of camera parameter box 407. Therefore, it is recorded in annotation 2 box 402. In this embodiment, the blur level information corresponding to annotation information 461 to 465 is recorded in annotation 2 box 402. However, it may be recorded in another box (for example, a box for a manufacturer-specific metadata recording area) together with blur level information 4073 generated at the time of shooting by camera 100.

[0071] In this embodiment, annotation information recorded by the camera 100 during image capture is recorded in the annotation 1 box, and annotation information added by a PC or the like after image capture is recorded in the annotation 2 box and subsequent boxes. Therefore, the device that recorded each piece of annotation information can be determined based on the metadata recording area in which the annotation information is recorded (whether or not it is recorded in the annotation 1 box). Other methods may be used to determine the device that recorded the annotation information. For example, information about the device that added the annotation information may be recorded in the annotation box. By comparing the camera model information included in the camera parameters with information about the device that added the annotation information, it becomes possible to determine whether each piece of annotation information was added by the camera that captured the image data 409.

[0072] 5A and 5B show display examples of image data 09 to which the metadata of FIG. 4B has been added on the PC 200. The image file can be displayed on the display unit 206 under the control of the image display control unit 304 and the annotation information display control unit 305, and in this case, annotation information can be displayed along with the image data 409. Regarding annotation information, it is possible to switch between displaying only annotation information recorded by the camera and displaying all annotation information recorded in the image file in response to a user's operation instruction. FIG. 5A shows an example in which, when an image file is displayed, only annotation information added by the camera 100 is displayed out of multiple pieces of annotation information contained in the image file. FIG. 5B shows an example in which, out of multiple pieces of annotation information contained in the image file, not only annotation information added by the camera 100 but also annotation information added by a device other than the camera 100 after the image was captured is displayed.

[0073] The image shown in FIG. 5A is an example of a display when a user captures HEIF image data 409 and displays the HEIF image file 400 having the structure of FIG. 4A on the camera 100.

[0074] Annotation information 452, 453, 454, and 455 added by the camera are recorded in the image file 400 in the structure of annotation 1' BOX 401, and the description of each annotation includes descriptions such as "face," "face," "mountain," and "river." Therefore, explanatory text information is displayed along with a frame indicating each annotation area. Furthermore, an AF frame 451 is also displayed based on the AF frame information 4072 in FIG. 4A. The AF frame and the frame indicating the annotation area are displayed in different display formats, such as by changing the color or the line style (solid line, dashed line, double line, etc.).

[0075] 5B , in addition to annotation information 452, 453, 454, and 455 added by the camera, annotation information 461, 462, 464, 464, and 465 added by the PC 200 are also displayed. The annotation descriptions for each of the annotation information 461, 462, 464, 464, and 465 include descriptions such as sun, face, cloud, cloud, face. Therefore, the explanatory text information is displayed along with a frame indicating each annotation area. In addition to annotation information 452, 453, 454, and 455 added by the camera, annotation information 461, 462, 464, 464, and 465 added by the PC 200 are displayed in a display format different from that of annotation information 452, 453, 454, and 455.

[0076] Next, the conversion process of the HEIF image file 400 in the PC 200 will be described with reference to Figures 6 and 7. In this process, the case where the HEIF image file 400 is converted into a JPEG image file will be described, but similar processes may be performed when converting into an image file in a format other than JPEG. The processes in Figures 6 and 7 are realized by the control unit (processing unit) 201 of the PC 200 controlling the components in Figures 2 and 3 based on a program read from the ROM 202.

[0077] First, the structure of a JPEG image file 500 will be described with reference to Figures 8A and 8B. A JPEG image file is basically defined to start with an SOI marker and end with an EOI marker, with metadata and image data included between these two markers. Therefore, the JPEG image file 500 is recorded in the following order: an SOI marker 5051, a 'meta1' segment 5061 (first recording area) in which metadata including annotation information can be recorded, image data 509, and an EOI marker 5081. The recordable size of the meta1' segment 5061 is predefined (prescribed) by the JPEG standard, and there is an upper limit to the data size. Therefore, in this embodiment, metadata that cannot be recorded in the meta1' segment 5061 is recorded in a 'meta2' segment 5062 (second recording area). When the 'meta2' segment 5062 is used, the SOI 5052, the 'meta2' segment 5062, and the EOI 5082 are recorded following the EOI 5081.

[0078] Because meta1 (first recording area) 5061 is also included in standard JPEG image files, it is a recording area in which annotations can be recorded and played back on both the camera 100 and the PC 200. In contrast, standard JPEG image files end with the first EOI 5081, meaning that the data thereafter, i.e., meta2 (second recording area) 5062, cannot be interpreted or played back on devices that can only play standard JPEG image files. For example, because the camera 100 only supports standard JPEG files, it cannot play back data in meta2 (second recording area) 5062, whereas the PC 200 can. Therefore, in the JPEG image file 500, meta1 (first recording area) 5061 is a recording area with higher playback compatibility than meta2 (second recording area) 5062.

[0079] Furthermore, on the PC, it is possible to set priority parameters in advance before starting the conversion process. Options are displayed on the display unit 206, and the user's selection of the parameters to be prioritized is accepted. The parameters selected by the user operating the operation unit 205 are set as priority parameters. The options displayed here include, for example, shooting mode information 4071, AF frame information 4072, and subject blur level information 4073, which are stored in the 'camera parameters' box 407.

[0080] 6 and 7 is initiated when a user selects a HEIF image file and priority parameters via the operation unit 205 of the PC 200 and issues an instruction to convert the selected HEIF image file into a JPEG image file. Before starting this process, the control unit 201 reads and acquires the image to be converted from the recording medium 204 using the image reading unit 300, and then acquires annotation information for the acquired image file using the annotation information acquisition unit 301.

[0081] 6 and 7 show the process of generating (recording) metadata into a JPEG image when converting a HEIF image file into a JPEG image file. When the metadata is generated by this process, the control unit 201 arranges the data so that it has the format shown in Figures 8A and 8B to generate a JPEG image file, and records the generated JPEG file as a new image file (new file) on the recording medium 204.

[0082] First, in step S100, the control unit 201 converts the camera parameters 407 of the HEIF image file 400 into the JPEG metadata recording format and stores them in the meta 1 segment (hereinafter referred to as the first recording area) 5061 of the JPEG image file 500.

[0083] In step S101, the control unit 201 determines whether or not the HEIF image file 400 contains annotation information added by the camera. If the HEIF image file 400 contains annotation information added by the camera, the process proceeds to step S102; if not, the process proceeds to step S103. The method for determining whether or not the camera contains annotation information is as described above. In this embodiment, if annotation information is recorded in the annotation 1 box 401, the process determines that the camera contains annotation information.

[0084] In step S102, the control unit 201 records the annotation information recorded by the camera 100, that is, the annotation information recorded in the annotation 1 box 401, in the first recording area, and then proceeds to step S103.

[0085] In step S103, the control unit 201 determines whether the HEIF image file 400 contains annotation information that was added by a device other than the camera, such as a PC (hereinafter also referred to simply as a PC). That is, the control unit 201 determines whether annotation information has been recorded in boxes from annotation 2 onwards. If annotation information that was added by a device other than the camera is present, the process proceeds to step S104; if not, the process ends the annotation information recording process.

[0086] In step S104, the control unit 201 determines whether there is remaining capacity in the first recording area of ​​the JPEG image file 500. If there is remaining capacity, that is, if the remaining capacity up to the upper limit size of the first recording area is equal to or greater than a predetermined capacity (size), the process proceeds to step S105;

[0087] In step S105, the control unit 201 acquires a preset parameter to be prioritized, and the process proceeds to step S106.

[0088] In step S106, the control unit 201 rearranges the annotation information added by the PC in the HEIF image file 400 according to the priority parameters acquired in step S105, and then proceeds to step S108. That is, based on the priority parameters, the order of priority of the annotation information to be recorded in the first recording area is determined and the information is rearranged in order of priority.

[0089] In step S108, the control unit 201 sequentially records the rearranged annotation information added by the PC one by one in the order in which it was arranged in the remaining first recording area of ​​the JPEG image file 500. Then, when the upper limit size of the first recording area is reached or the remaining capacity up to the upper limit falls to a predetermined value or less, recording of the annotation information in the first recording area is terminated at that point, and the process proceeds to step S109. When the upper limit size of the first recording area is reached, the annotation information most recently recorded in the first recording area is deleted, and the process proceeds to step S109.

[0090] By this process, annotation information added by the PC is preferentially recorded in the first recording area in the order sorted based on the priority parameters (priority order). Annotation information with a low priority order is more likely to be recorded in the second recording area rather than the first recording area.

[0091] In step S109, the control unit 201 determines whether all of the annotation information added by the PC has been recorded in the first recording area. If all of the annotation information has been recorded, the process ends. If not all of the annotation information has been recorded, the process proceeds to step S110.

[0092] In step S110, the control unit 201 records the annotation information added by the remaining PCs in the second recording area (meta2 segment 5062) of the JPEG image file 500, and then ends this processing.

[0093] Next, specific processing of S105 to S107 when the parameters prioritized in step S100 are the shooting mode information 4071, the AF frame information 4072, and the subject blur level information 4073 in that order will be described with reference to FIG.

[0094] It is assumed that HEIF image file 400 has annotation information 452, 453, 454, and 455 added by the camera and annotation information 461, 462, 463, 464, and 465 added by the PC, as shown in FIG. 4B.

[0095] Shooting mode information 4071 stores "Portrait," and AF frame information 4072 stores information about AF frame 451. Subject blur level information 4073 indicates that annotation information 452 to 455 added by the camera is judged as good, annotation information 461, 462, and 463 added by the PC is judged as average, and the remaining annotation information 464 and 465 is judged as good. Here, a good judgment indicates that the degree of blur is less than an average judgment.

[0096] Also, in step 104, it is assumed that the first recording area of ​​the JPEG image file 500 has a remaining capacity that can record three pieces of annotation information.

[0097] In step S1051, the control unit 201 acquires the information "portrait" from the shooting mode information 4071, and proceeds to step S1061.

[0098] In step S1061, the control unit 201 determines the first priority of each of the annotation information 461, 462, 463, 464, and 465 added by the PC based on the shooting mode information 4071, and proceeds to step S1062. In this embodiment, since the mode is "portrait," priority is given to information related to people. Since the description of the annotation information 462 and 465 added by the PC is "face," the first priority is set to high, followed by the first priority of the annotation information 461, 463, and 464 added by the PC, which is set to medium.

[0099] In step S1062, the control unit 201 rearranges the annotation information 461, 462, 463, 464, and 465 added by the PC in descending order, taking into consideration the priority of each piece of annotation information. Then, the process proceeds to step S1071. In this embodiment, the order of the annotation information added by the PC is 462, 465, 461, 463, and 464.

[0100] In step S1071, the control unit 201 determines whether any of the annotation information 461, 462, 463, 464, and 465 added by the rearranged PC has the same priority. If so, the process proceeds to step S1052; if not, the process proceeds to step S108 shown in FIG. 6. In this embodiment, since there is an identical piece of annotation information with the first priority, the process proceeds to step S1052.

[0101] In step S1052, the control unit 201 acquires the position information of the AF frame 451 from the AF frame information 4072, and the process proceeds to step S1063.

[0102] In step S1063, the control unit 201 determines the second priority of each of the annotation information 461, 462, 463, 464, and 465 added by the PC based on the position information of the AF frame 451, and proceeds to step S1064. In this embodiment, the second priority of the annotation information 462 added by the PC, which overlaps with the position information of the AF frame 451, is set to high, and the second priorities of the remaining four are set to medium.

[0103] In step S1064, the control unit 201 rearranges the annotation information 461, 462, 463, 464, and 465 added by the PC in descending order, taking into consideration the first priority and second priority of the annotation information 461, 462, 463, 464, and proceeds to step S1072. In this embodiment, the order of the annotation information 462, 465, 461, 463, and 464 added by the PC is used.

[0104] In step S1072, the control unit 201 determines whether any of the annotation information 461, 462, 463, 464, and 465 added by the rearranged PC has the same first and second priorities. If so, the process proceeds to step S1053; if not, the process proceeds to step S108 shown in FIG. 6. In this embodiment, the annotation information 461, 463, and 464 have the same first and second priorities, so the process proceeds to step S1053.

[0105] In step S1053, the control unit 201 acquires the blur level information 4073 of the subject, and the process proceeds to step S1065.

[0106] In step S1065, the control unit 201 determines the third priority of each of the annotation information 461, 463, and 464 added by the PC in the HEIF image file 400 based on the blur degree information 4073 of the subject, and proceeds to step S1066. In this embodiment, the annotation information 464 added by the PC, which is determined to be good in terms of the blur degree, is assigned a high third priority, and the annotation information 461 and 463 added by the PC, which is determined to be fair, are assigned a medium third priority.

[0107] In step S1066, the control unit 201 rearranges the annotation information 461, 462, 463, 464, and 465 added by the PC in descending order, taking into consideration the first priority, second priority, and third priority, and proceeds to step S108 in Fig. 6. In this embodiment, the order of priority of the annotation information added by the PC is 462, 465, 464, 461, and 463. Here, the order in which the annotation information is recorded is maintained for those with the same priority, but other processing may be added to rearrange the information.

[0108] In step S108, the first recording area has enough capacity to record three pieces of annotation information, so the annotation information 462, 465, and 464 added by the PC are recorded in the first recording area. As a result, the answer in step S109 is No, and the annotation information 461 and 463 added by the PC are recorded in the second recording area by the processing in step S110.

[0109] As a result, as shown in FIG. 8A, which will be described later, a JPEG image file 500 is generated that includes the annotation information added by the camera 100 and the PC 200.

[0110] In the following description, the 'annotation 1' box 401 corresponds to the annotation information 452, 453, 454, and 455 added by the camera, as shown in FIG. 4B.

[0111] Furthermore, 'annotation 2' box 402, 'annotation 3' box 403, and 'annotation 4' box 404 are annotation information 461, 462, 463, 464, and 465 added by the PC, as shown in Fig. 4B. In particular, 'annotation 2' box 402 is an annotation determined to have a high priority by a program described below, and corresponds to annotation information 462, 464, and 465 added by the PC in Fig. 5B. 'annotation 3' box 403 and 'annotation 4' box 404 are annotations determined to have a low priority by a program described below, and correspond to annotation information 461 and 463 added by the PC in Fig. 5B.

[0112] FIG. 8A shows an example of a JPEG image file generated by performing the JPEG conversion process of this embodiment on the HEIF image file 400 of FIG. 4A.

[0113] A marker SOI 5051 indicating the start of the image file is recorded at the beginning of the JPEG image file 500 , and a 'meta1' segment 5061 is recorded in the line following the marker SOI 5051 .

[0114] The recordable capacity of the 'meta1' segment 5061 (first recording area) is smaller than the recordable capacity of the 'meta' box 406 in the HEIF image file 400. In the 'meta1' segment 5061, an area 5011 for annotation 1, a description 5012 for annotation 1, an area 5021 for annotation 2, and a description 5022 for annotation 2 are recorded.

[0115] The annotation 1 area 5011 records information similar to that in the annotation 1 area of ​​the HEIF image file 400. The annotation 1 description 5012 records information similar to that in the annotation 1 description 4012 in the HEIF image file 400. The annotation 2 area 5021 records information similar to that in the annotation 2 area of ​​the HEIF image file 400. The annotation 2 description 5022 records information similar to that in the annotation 2 description 4022 in the HEIF image file 400.

[0116] Image data 509 is recorded after the description 5022 of annotation 2. The image data 509 is image data obtained by converting the image data 409 in the HEIF image file into JPEG format. A marker EOI 5081 indicating the end of the image data is recorded after the image data 509. A marker SOI 5052 indicating the second recording area is recorded on the line following the marker SOI 5081, and a 'meta2' segment 5062 is recorded on the line following the marker SOI 5052.

[0117] The 'meta2' segment 5062 (second recording area) is a recording area with no capacity limit, unlike the 'meta1' segment 5061. In the 'meta2' segment 5062, an area 5031 for annotation 3, a description 5032 for annotation 3, an area 5041 for annotation 4, and a description 5042 for annotation 4 are recorded.

[0118] Annotation 3 field 5031 records information similar to that in annotation 3 field 4031 in the HEIF image file 400. Annotation 3 description 5032 records information similar to that in annotation 3 description 4032 in the HEIF image file 400. Annotation 4 field 5041 records information similar to that in annotation 4 field 4041 in the HEIF image file 400. Annotation 4 description 5042 records information similar to that in annotation 4 description 4042 in the HEIF image file 400.

[0119] After the explanation 5042 of annotation 4, a marker EOI 5082 indicating the end of the second recording area is recorded.

[0120] Compared to image files in other formats such as HEIF, the JPEG image file 500 has a smaller recording capacity in the area for recording annotation information ('meta1' segment 5061). For this reason, when converting a HEIF image file or the like to a JPEG image file, the annotation information recorded in the image file before conversion cannot be recorded as is in the JPEG image file. As a result, although playback compatibility will be reduced, annotation information that cannot be recorded in the 'meta1' segment 5061 (first recording area) is recorded in the 'meta2' segment 5062 (second recording area).

[0121] In an embodiment of the present invention, when JPEG image file 500 is displayed on an image processing device that can only play back the first recording area, such as camera 100, it is displayed as shown in Fig. 8B. Specifically, annotation information 452, 453, 454, and 455 added by the camera and annotation information 462, 465, and 464 added by a PC other than the camera are displayed. However, on a device that can play back the second recording area, such as PC 200, all annotation information can be displayed.

[0122] Although the present invention has been described above using the above-mentioned embodiment, the present invention is not limited to the above-mentioned embodiment.

[0123] In an embodiment of the present invention, all annotation information is recorded in a single JPEG image file 500, as shown in FIG. 8A, but it may also be written out and recorded in a different file format as a second recording area.

[0124] Furthermore, annotation information that cannot be recorded in the format of the first recording area may be written in the second recording area. For example, consider the case of a HEIF image file 600 shown in Figure 9. An 'annotation 1' box 601 and an 'annotation 2' box 602 are added to the HEIF image file 600 by the camera. An 'annotation 3' box 603 and an 'annotation 4' box 604 are also added in a recording format different from that of the 'annotation 1' box 601 and the 'annotation 2' box 602.

[0125] More specifically, the 'annotation 3' box 603 and the 'annotation 4' box 604 record language information 6033 for annotation 3 and language information 6043 for annotation 4. This differs from the HEIF image file 400 in Fig. 4A. The language information 6033 for annotation 3 and the language information 6043 for annotation 4 store the language of the country in which the description 6032 for annotation 3 and the description 6042 for annotation 4 are written; for example, in the case of Japanese, "JP" is recorded.

[0126] 10A and 10B , if the data format of meta1 (first storage area) 7061 of the converted JPEG image file 700, which includes language information, is not supported by the camera 100, the language information may be recorded in meta2 (second storage area) 7062. That is, the 'annotation 3' box 603 and the 'annotation 4' box 604 are recorded in the second storage area. The language information 6033 of annotation 3 and the language information 6043 of annotation 4 may be recorded in the second storage area as the language information 7033 of annotation 3 and the language information 7043 of annotation 4, respectively.

[0127] This allows annotations to be recorded in an area with high playback compatibility and in a standard with high playback compatibility, so that the annotations intended by the user are recorded in an area with high playback compatibility, making it possible to play back the annotation information intended by the user on other playback devices.

[0128] Although the PC 200 has been described as an example of a device other than a camera that adds annotation information, a smartphone or other device may also be used. Also, in the embodiment of the present invention, annotation information is added using an image editing application on the PC 200, but annotation information may be added in another form, such as a web browser, instead of an image processing application.

[0129] Furthermore, the area information of the annotation does not have to be rectangular, and may be any shape, such as a polygon or ellipse, as long as it can indicate an area.

[0130] Although the conversion from HEIF to JPEG has been described, it can also be applied to image editing, such as image compositing. In particular, in the case of image compositing, it is possible to give priority to the annotation information of the main image and record it in the first recording area.

[0131] In the embodiment of the present invention, the effect is described that the annotation information intended by the user can be reproduced on the camera (first device), but it may also be possible to view it on other general-purpose devices.

[0132] In the embodiment of the present invention, the user selects the order of priority, but the image processing application and control unit 201 of the PC 200 may decide which parameters are given priority.

[0133] Although the camera parameters for shooting are used, the annotation information itself may also be used. For example, the options displayed in S100 may include the size of the area 4011 of annotation 1, the type of explanation 4012 of annotation 1, and the like, contained in the 'annotation 1' box 401.

[0134] In the embodiment of the present invention, the priority of the explanation of the annotation information for each piece of shooting mode information may be represented by a table, or the relationship between the shooting mode and the explanation of the annotation information may be quantified.

[0135] In the embodiment of the present invention, for simplicity, the annotation information area and the blur degree determination area are the same, but they may be different. For example, if there are multiple blur degree determination areas that overlap with the annotation information area, the priority of the corresponding annotation information may be calculated arithmetically, such as by taking the average of the multiple blur degrees, or the best or lowest value. Similarly, other priority determination methods may be calculated arithmetically for stored values.

[0136] In step S102, the annotation information recorded by the camera 100 is written on the assumption that it will not overflow from the first recording area of ​​the JPEG image file 500, but if it does overflow, the process may proceed to step S110.

[0137] Furthermore, the annotation information recorded in the first recording area may be recorded in the second recording area in an overlapping manner.

[0138] In the embodiment of the present invention, annotation information is recorded by a camera and a PC, but annotations may be added by a device other than the PC. Also, a determination unit for determining which device added the annotation may be provided. In this case, the determination may be made from the annotation itself, or from camera information or a maker note included in the image file. In this case, a table indicating the correspondence between the camera, PC, and another device may be prepared in advance, or a web search may be performed to determine the correspondence. Furthermore, if an annotation cannot be determined, it may be treated as an unknown annotation.

[0139] For example, if an image file contains camera annotations, PC annotations, and unknown annotations, the user may be allowed to select whether to prioritize the PC annotations or the unknown annotations. The priority may also be determined by the PC 200. Both annotations may also be treated as having the same priority.

[0140] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0141] Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0142] <Other Embodiments> The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and one or more computers (or CPUs, MPUs, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. The present invention can also be realized by a circuit that realizes one or more functions.

[0143] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0144] This application claims priority based on Japanese Patent Application No. 2024-033089, filed March 5, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: an acquisition means for acquiring an image file containing multiple pieces of annotation information; and a control means for controlling the generation and recording of a new image file based on the image file acquired by the acquisition means, wherein the control means controls the recording of annotation information added to the image file by an imaging device in a first recording area of ​​the new image file, and the recording of annotation information added to the image file by a device other than the imaging device in a second recording area of ​​the new image file, the second recording area having lower playback compatibility than the first recording area.

2. The imaging device according to claim 1, characterized in that the acquisition unit acquires an image file in HEIF format, and the control means controls the recording of an image file in JPEG format generated based on the image file in HEIF format.

3. An image processing device as described in claim 1 or 2, characterized in that the first recording area is the area between the SOI, which is the beginning of the JPEG file, and the next EOI, and the second recording area is the area after the EOI.

4. An image processing device as described in any one of claims 1 to 3, characterized in that the control means processes annotation information stored in a specific box among the multiple annotation information contained in the image file as annotation information added by the imaging device, and processes annotation information stored in a box other than the specific box as annotation information added by a device other than the imaging device.

5. An image processing device as described in any one of claims 1 to 4, characterized in that the first recording area has a specified upper limit on the recordable data size, and the control means controls the recording of annotation information added by a device other than the imaging device in the first recording area if the size of the first recording area does not reach the upper limit after storing annotation information added by the imaging device.

6. The image processing device described in claim 5, characterized in that the control means controls annotation information added by a device other than the imaging device to be recorded in the first recording area within a range not exceeding the upper limit, and determines the annotation information to be recorded in the first recording area based on a predetermined priority.

7. An image processing device as described in claim 6, further comprising a selection means for selecting priority parameters, wherein the control means determines the annotation information to be recorded in the first recording area with a priority according to the priority parameters selected by the selection means.

8. The image processing device according to claim 7, wherein said control means controls so that annotation information related to the priority parameter selected by said selection means is preferentially recorded in said first recording area.

9. An image processing device as described in any one of claims 6 to 8, characterized in that the control means controls annotation information added by a device other than the imaging device that cannot be recorded in the first recording area to be recorded in the second recording area.

10. A control method for an image processing device, comprising: an acquisition step of acquiring an image file containing multiple pieces of annotation information; and a control step of controlling to generate a new image file based on the image file acquired in the acquisition step, wherein in the control step, of the multiple pieces of annotation information contained in the image file, annotation information added to the image file by an imaging device is controlled to be recorded in a first recording area of ​​the new image file, and annotation information added to the image file by a device different from the imaging device is controlled to be recorded in a second recording area of ​​the new image file, which is a second recording area having lower playback compatibility than the first recording area.

11. A program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 9.

12. A program for causing a computer to execute the control method for an image processing apparatus according to claim 10.

Images