Image processing apparatus, control method thereof, storage medium, and computer program product

Abstract

The invention provides an image processing apparatus, a control method thereof, a storage medium, and a computer program product. An image processing apparatus includes: an acquisition unit configured to acquire an image file including a plurality of annotations; a determination unit configured to determine a correlation between the plurality of annotations; and a control unit configured to perform control to integrate the annotations determined as relevant by the determination unit among the plurality of annotations into an integrated annotation and output the integrated annotation.

Description

Technical Field

[0001] The present invention relates to an image processing device for processing an image file of an image captured by a camera or the like. Background Art

[0002] In recent years, artificial intelligence (AI) technologies such as deep learning have been used in various technological fields.

[0003] For example, existing digital cameras and the like have a function of detecting human faces from captured images. Japanese Patent Application Laid-Open No. 2015-99559 discloses a technology for accurately detecting and recognizing humans and animals such as dogs or cats.

[0004] A technique for recording information about a detected object as an annotation in association with image data has also been proposed. For example, it is also considered to record information indicating the area of ​​the detected object as an annotation using a point, rectangle, circle, etc.

[0005] There are various standards for recording annotations associated with image data. Furthermore, even with a single image format, multiple annotations conforming to different standards can be recorded. Japanese Patent Application Laid-Open No. 2023-173960 proposes a method for recording annotations when converting an image between different image formats.

[0006] Because the accuracy of subject detection varies between recording devices that record annotations, such as digital cameras and image processing software in personal computers (PCs), recording devices that detect the same subject (object) will not record exactly the same annotations. For this reason, when recording annotations in a single image file using multiple recording devices, multiple different annotations may be recorded for the same subject (object). Summary of the Invention

[0007] The present invention provides an image processing apparatus that allows a user to easily process a plurality of annotations recorded in an image file.

[0008] According to one aspect of the present invention, an image processing device is provided, which includes: an acquisition unit, which is configured to acquire an image file including multiple annotations; a determination unit, which is configured to determine the correlation between the multiple annotations; and a control unit, which is configured to control to integrate the annotations among the multiple annotations determined by the determination unit to be relevant into an integrated annotation and output the integrated annotation.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1is a block diagram illustrating an example configuration of a digital camera according to an embodiment.

[0011] Figure 2 is a block diagram showing an example configuration of a PC according to the embodiment.

[0012] Figure 3 is a block diagram illustrating a processing unit of a digital camera or a PC according to an embodiment.

[0013] Figure 4 is a block diagram illustrating an example configuration of an image file according to the embodiment.

[0014] Figure 5A and Figure 5B is a block diagram illustrating an example configuration of an annotation according to an embodiment.

[0015] Figure 6A and Figure 6B is a diagram illustrating a specific example of annotation according to the embodiment.

[0016] Figure 7 is a block diagram showing a processing unit according to the first embodiment.

[0017] Figure 8 is a flowchart illustrating the annotation integration process according to the first embodiment.

[0018] Figure 9 An example of annotations before the annotation integration process according to the first embodiment is performed is shown.

[0019] Figure 10 An example of annotations after the annotation integration process according to the first embodiment is shown.

[0020] Figure 11 is a block diagram showing a processing unit according to the second embodiment.

[0021] Figure 12A and Figure 12B An example of annotation of an image file according to the second embodiment is shown.

[0022] Figure 13A 、 Figure 13B 、 Figure 13C 、 Figure 13D 、 Figure 13E and Figure 13F A display example of annotations according to the second embodiment is shown.

[0023] Figure 14 is a flowchart illustrating layer reconstruction processing according to the second embodiment.

[0024] Figure 15 : is a flowchart showing the comment inclusion relationship determination process in the layer reconstruction process according to the second embodiment.

[0025] Figure 16 is a flowchart illustrating an adding process in the layer reconstruction process according to the second embodiment.

[0026] Figure 17A 、 Figure 17B 、 Figure 17C and Figure 17D : is a diagram showing annotations obtained when performing the layer reconstruction process according to the second embodiment.

[0027] Figure 18A 、 Figure 18B 、 Figure 18C and Figure 18D : is a diagram showing annotations obtained when performing the layer reconstruction process according to the second embodiment.

[0028] Figure 19 A display example of annotations obtained when performing the layer reconstruction process according to the second embodiment is shown.

[0029] Figure 20 is a flowchart illustrating image display processing according to the third embodiment.

[0030] Figure 21 : is a flowchart showing an annotation display process in the image display process according to the third embodiment.

[0031] Figure 22 is a flowchart illustrating a modification of the image display process according to the third embodiment.

[0032] Figure 23A 、 Figure 23B 、 Figure 23C and Figure 23D A display example of annotations according to the third embodiment is shown.

[0033] Figure 24A and Figure 24B A display example of annotations in an enlarged view according to the third embodiment is shown.

[0034] Figure 25A and Figure 25B is a diagram illustrating an example of annotation according to the fourth embodiment.

[0035] Figure 26 is a block diagram illustrating a processing unit according to the fourth and fifth embodiments.

[0036] Figure 27 is a flowchart illustrating a sorting process according to the fourth embodiment.

[0037] Figure 28 is a flowchart illustrating a method of determining a reference based on reliability in a ranking process according to the fourth embodiment.

[0038] Figure 29 An example of annotation according to the fifth embodiment is shown.

[0039] Figure 30 is a flowchart illustrating filtering processing according to the fifth embodiment.

[0040] Figure 31 is a flowchart illustrating a method of determining a reference based on reliability in filtering processing according to the fifth embodiment. DETAILED DESCRIPTION

[0041] The following embodiments are described in detail with reference to the accompanying drawings. The following embodiments do not limit the claimed invention. Although the embodiments describe multiple features, not all of these features are essential to the invention, and multiple features may be used in any combination. In addition, in the drawings, identical or similar components are denoted by the same reference numerals, and their repeated description will be omitted.

[0042] System Structure

[0043] In the following description, a camera (imaging device) and a PC (information processing device) are provided as examples of image processing devices. However, the image processing device according to an embodiment of the present invention is not limited to these examples. The image processing device according to an embodiment of the present invention may be any image processing device that reads an image recorded on a recording device (recording medium) and displays the read image on a display device or outputs the read image as an image file. Examples of image processing devices include smartphones and tablet PCs.

[0044] Figure 1 is a block diagram illustrating an example configuration of a digital camera 100 according to an embodiment of the present invention.

[0045] The protective cover 10 is a protective member that covers the imaging unit (including the camera lens 11) of the digital camera 100 to protect the imaging unit from dust or damage. The camera lens 11 forms an optical image on the imaging surface of the imaging element 13. The shutter 12 has an aperture function. The imaging element 13 converts the optical image formed on the imaging surface into an electrical signal. The imaging element 13 includes, for example, a charge coupled device (CCD) element, a complementary metal oxide semiconductor (CMOS) element, etc. The analog-to-digital (A / D) converter 15 converts the analog signal output from the imaging element 13 into a digital signal. The digital signal obtained by the A / D converter 15 through conversion is written into the memory 25 as so-called raw image data. In addition, development parameters corresponding to each raw image data are generated based on the information obtained during imaging, and the development parameters are written into the memory 25. The development parameters include various parameters used in image processing for recording Joint Photographic Experts Group (JPEG) data, etc., such as exposure settings, white balance, color space, and contrast.

[0046] The timing generator 14 supplies clock signals and control signals to the imaging element 13, the A / D converter 15, and the digital-to-analog (D / A) converter 21. The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50A. The image processing unit 20 performs various types of image processing, such as predetermined pixel interpolation processing, color conversion processing, correction processing, and resizing processing, on the data from the A / D converter 15 or the data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and arithmetic processing using the captured image data and supplies the obtained arithmetic results to the system control unit 50A. Based on the supplied arithmetic results, the system control unit 50A controls the exposure control unit 40 and the distance measurement control unit 41 to implement autofocus (AF) processing, automatic exposure (AE) processing, and pre-flash (EF) processing.

[0047] The image processing unit 20 performs predetermined arithmetic processing using the captured image data and also performs automatic white balance (AWB) processing based on the obtained arithmetic results. Furthermore, the image processing unit 20 reads images stored in the memory 25 and performs compression, such as lossless compression of uncompressed raw data, and compression or decompression processing in accordance with the JPEG, MPEG-4 Advanced Video Coding (AVC), or High Efficiency Video Coding (HEVC) standards. The image processing unit 20 writes the processed data to the memory 25.

[0048] The image processing unit 20 also performs predetermined arithmetic processing using the captured image data, and performs processing for editing various types of image data. Specifically, the image processing unit 20 is capable of performing a cropping process for hiding unnecessary peripheral portions of the image data to adjust the display range and size of the image, and a resizing process for enlarging or reducing image data, elements displayed on the screen, etc. to change the size. In addition, the image processing unit 20 performs the following raw (RAW) development, which is used to apply image processing such as color conversion to data that has undergone compression processing (such as lossless compression of uncompressed raw data) or decompression processing, and convert the resulting data into JPEG data to create image data. In addition, the image processing unit 20 is capable of implementing a video cropping process for cropping a specified frame of a video format such as the MPEG-4 format and converting and storing the frame in the JPEG format.

[0049] The image processing unit 20 also performs processing for, for example, superimposing an on-screen display (OSD) such as a menu or any characters to be displayed on the display unit 23 described below together with the display image data on the display image data.

[0050] The image processing unit 20 also performs subject detection processing for detecting a subject present in the input image data using image data, distance information to the subject obtained from the imaging element 13 during image capture, etc., and detecting the subject's area. Thus, information about the area, such as the position and size of the area in the image, and detection information, such as inclination and certainty, can be obtained as detectable information.

[0051] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The raw image data generated by the A / D converter 15 is written to the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or via the memory control unit 22 without the intervention of the image processing unit 20.

[0052] The image display memory 24, the D / A converter 21, and the thin-film transistor (TFT) liquid crystal display (LCD) constitute the display unit 23. Image data for display written to the image display memory 24 is displayed on the display unit 23 via the D / A converter 21. Using the display unit 23 to sequentially display captured image data enables an electronic viewfinder function for displaying real-time images. The memory 25 stores captured still images and moving images. The memory 25 has a storage capacity sufficient to store a predetermined number of still images and a predetermined length of moving images. The memory 25 can also serve as a workspace for the system control unit 50A.

[0053] The exposure control unit 40 controls the shutter 12, which has an aperture function. Furthermore, the exposure control unit 40 has a flash dimming function by operating in conjunction with the flash 44. The distance measurement control unit 41 controls the focus of the imaging lens 11. The zoom control unit 42 controls the zoom of the imaging lens 11. The protective cover control unit 43 controls the operation of the protective cover 10, which serves as a protective member. The flash 44 has an AF auxiliary light projection function and a flash dimming function.

[0054] The system control unit 50A controls the entire digital camera 100. The nonvolatile memory 51 is an electrically erasable and recordable nonvolatile memory. Examples of the nonvolatile memory 51 include an electrically erasable programmable read-only memory (EEPROM). The nonvolatile memory 51 stores not only programs but also map information and the like. The system control unit 50A also functions as a processing unit that executes processing according to the programs stored in the nonvolatile memory 51. Figure 3 Describe the processing unit in detail.

[0055] When the shutter button 60 is operated halfway, the shutter switch 61 (SW1) turns on and issues an instruction to start an operation such as AF processing, AE processing, AWB processing, or EF processing. When the shutter button 60 is fully operated, the shutter switch 62 (SW2) turns on and issues an instruction to start a series of imaging operations, which include exposure processing, development processing, and recording processing. In the exposure processing, the signal read from the imaging element 13 is written as raw data to the memory 25 via the A / D converter 15 and the memory control unit 22. In the development processing, the raw data written to the memory 25 is developed using calculations performed by the image processing unit 20 or the memory control unit 22 and written as image data to the memory 25. In the recording processing, the image data is read from the memory 25 and compressed by the image processing unit 20. The compressed image data (image file) is stored in the memory 25 and then written to the external recording medium 91 via the card controller 90.

[0056] The operation unit 63 includes various buttons, a touch panel, and the like. The operation unit 63 includes, for example, a power button, a menu button, a mode switch for switching between modes (i.e., image capture mode, playback mode, and other special image capture modes), a cross key, a set button, a macro button, and multi-screen playback paging buttons. The operation unit 63 also includes, for example, a flash setting button, a button for switching between single-shot, continuous, and Selfie modes, a menu movement plus (+) button, a menu movement minus (-) button, a button for selecting image quality, an exposure compensation button, a date / time setting button, and the like.

[0057] To record image data as an image file on the external recording medium 91, the metadata generation and analysis unit 70 generates various metadata for the image data that conforms to standards such as the Exchangeable Image File (Exif) format based on information obtained during imaging. Furthermore, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata assigned to the image data. Examples of metadata include imaging setting information regarding settings during imaging, image data information associated with the image data, subject identification information (depth information, subject features, etc.), and annotations. To record moving image data, the metadata generation and analysis unit 70 can generate metadata for each frame and assign the metadata to the moving image data. In the digital camera 100 according to this embodiment, when the system control unit 50A performs an imaging operation and records an image file, the metadata generation and analysis unit 70 generates metadata as described above, and the system control unit 50A stores the metadata in the metadata recording area of ​​the image file. The system control unit 50A then controls the card controller 90 to record the generated image file on the external recording medium 91.

[0058] The power supply 80 includes, for example, a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a lithium battery, and an AC adapter. The power supply control unit 81 supplies power supplied from the power supply 80 to each unit of the digital camera 100. The card controller 90 sends and receives data to an external recording medium 91 such as a memory card. The external recording medium 91 (an example of which includes a memory card) stores images (still images or moving images) captured by the digital camera 100.

[0059] Inference engine 73A uses the inference model stored in inference model recording unit 72A to perform inference on image data input from system control unit 50A. The inference model may be an inference model input from external device 101 or the like via communication unit 71A and recorded in inference model recording unit 72A, or may be an inference model newly trained by learning unit 74A, described below. Inference engine 73A includes a neural network design 73a.

[0060] Neural network design 73a includes an input layer, an output layer, and an intermediate layer (neurons) between the input and output layers. The input layer receives image data input from system control unit 50A. The intermediate layer includes several layers of neurons. The number of neuron layers is appropriately determined by the design, and the number of neurons in each layer is also appropriately determined by the design. The intermediate layers are weighted based on the inference model recorded in inference model recording unit 72A. The output layer outputs annotations corresponding to the image input to the input layer.

[0061] The inference engine 73A can be updated from the external device 101 or the like, and can process various inference models.

[0062] The learning unit 74A will be described below. The learning unit 74A retrains the inference model in response to a request from the system control unit 50A or the like. The learning unit 74A includes a training data recording unit 74a. The training data recording unit 74a records information about the training data of the inference engine 73A. The learning unit 74A can retrain the inference engine 73A using the training data recorded in the training data recording unit 74a and update the inference engine 73A using the inference model recording unit 72A.

[0063] The inference model recording unit 72A and the like manage the management versions of the inference models so that each inference model can be identified when the inference model is updated from the outside or when the inference model is retrained and updated by the learning unit 74A.

[0064] In this embodiment, the inference engine 73A performs inference using a neural network. However, any method capable of classifying, inferring, and analyzing a subject included in an image may be used.

[0065] The communication unit 71A includes a communication circuit for transmitting and receiving. Transmission and reception can be performed via wireless communication such as Wi-Fi or Bluetooth, or wired communication such as Ethernet or a universal serial bus (USB). The communication unit 71A is capable of communicating with the communication unit 71B of the external device 101. The communication unit 71A functions as a communication unit that not only transmits and receives image files assigned annotations created by the inference engine 73A to and from the system control unit 50A and the system control unit 50B, but also transmits and receives various types of information, such as inference models and training data, to and from the system control unit 50A and the system control unit 50B. The information to be transmitted can be limited depending on whether the external device 101 is associated with the digital camera 100.

[0066] The external device 101 includes a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system control unit 50B, and a communication unit 71B. Alternatively, the external device 101 may be a device that does not include these components. The learning unit 74B creates an inference model in response to external requests received from the inference engine 73B, the system control unit 50B, etc. The inference model recording unit 72B records therein the inference model transmitted from the digital camera 100 and the inference model created by the learning unit 74B.

[0067] Figure 2 is a block diagram showing an example configuration of the PC 200 according to the embodiment of the present invention.

[0068] The control unit 201 is, for example, a central processing unit (CPU), and controls the entire PC 200. The control unit 201 also functions as a processing unit that executes processing according to a program stored in a read-only memory (ROM) 202. Figure 3 Describe the processing unit in detail.

[0069] The ROM 202 stores programs and parameters in a non-transitory manner.

[0070] A random access memory (RAM) 203 temporarily stores programs and data supplied from an external device or the like.

[0071] The recording medium 204 is a hard disk or a flash memory fixedly mounted in the PC 200 , or a medium removable from the PC 200 such as an optical disk, a magnetic card, an optical card, an integrated circuit (IC) card, or a memory card.

[0072] The operation unit 205 receives a user operation on the PC 200. The operation unit 205 has a function of selecting one of the still images displayed on the display unit 206 according to an operation performed by the editor and satisfying a predetermined condition. The user can perform an operation using an operation member such as a button or a touch panel provided in the PC 200 or using an operation member such as a keyboard or a mouse detachable from the PC 200.

[0073] The display unit 206 displays data stored in the PC 200 , data supplied externally, etc. The display unit 206 has a function of further displaying metadata assigned to an image recorded on the recording medium 204 .

[0074] The communication unit 207 communicates with an external device such as the digital camera 100 .

[0075] The system bus 208 communicatively connects the components of the PC 200 to each other.

[0076] The PC 200 can communicate with the communication unit 71A of the digital camera 100 via the communication unit 207 to acquire an image file from the digital camera 100 and display the acquired image file on the display unit 206. In addition, the control unit 201 can generate a new image file based on the acquired image file and record it on the recording medium 204. The acquisition of the image file can be performed not via the communication unit 207 but via the recording medium 204. Specifically, the external recording medium 91 that stores the image files captured by the digital camera 100 can be attached to the PC 200 as the recording medium 204.

[0077] Figure 3 1 is a block diagram illustrating the configuration of a processing unit included in the system control unit 50A of the digital camera 100 and a processing unit included in the control unit 201 of the PC 200 .

[0078] The system control unit 50A of the digital camera 100 and the control unit 201 of the PC 200 also function as processing units by loading predetermined programs and the like stored in the nonvolatile memory 51 and the ROM 202, respectively, into the work RAM and executing the programs. Figure 3 The units 300 to 306 included in the processing unit (the system control unit 50A or the control unit 201) will be described.

[0079] The image reading unit 300 reads an image file from a storage medium and acquires the image file. The image file may be acquired via the communication unit 71A or 207.

[0080] The comment acquisition unit 301 acquires the comment recorded in the image file from the image file read by the image reading unit 300 .

[0081] The image storage unit 302 performs processing for storing (recording) the image file generated by the processing unit in a storage medium.

[0082] The annotation assigning unit 303 assigns an annotation to the image file newly stored in the storage medium by the image storage unit 302. In this embodiment, the annotation to be assigned is recorded in the metadata area of ​​the image file. However, the annotation can also be recorded as a separate file related to the image file.

[0083] The image display control unit 304 performs processing for displaying image data of an image file read by the image reading unit 300 or an image file generated by the processing unit on the display unit 23 or 206 .

[0084] The annotation display control unit 305 performs the following processing: displays the annotation acquired by the annotation acquisition unit 301 or the annotation to be assigned to the image file by the annotation assignment unit 303 on the display unit 23 or 206. The annotation display control unit 305, in conjunction with the image display control unit 304, can display the image data and annotation of the image file on the display unit 23 or 206 in a superimposed manner, or display the image data and annotation separately on the display unit 23 or 206.

[0085] The annotation generation unit 306 detects the subject and generates an annotation using the inference engine 73A or 73B in the digital camera 100 or a predetermined inference program in the PC 200. Since the digital camera 100 includes the metadata generation and analysis unit 70, the annotation generation unit 306 can be omitted, and the metadata generation and analysis unit 70 can generate metadata.

[0086] Figure 4 is a block diagram illustrating an example configuration of an image file according to an embodiment of the present invention.

[0087] The image file 400 includes a metadata recording area 401 and an image area 402 for storing image data.

[0088] The metadata recording area 401 includes imaging setting information 403 , a comment recording area A 404 , and a comment recording area B 405 .

[0089] The imaging setting information 403 includes, for example, setting information recorded by the digital camera 100 when imaging, image data information related to the image data 402 , and subject recognition information.

[0090] The annotation recording area A 404 and the annotation recording area B 405 are areas for recording annotations. In this embodiment, the annotations recorded in the annotation recording area A 404 and the annotation recording area B 405 are based on different standards.

[0091] The comment recording area A 404 stores comments 406 acquired when the digital camera 100 captures the image data 402. The comment recording area B 405 stores comments generated based on the image data 402 and the imaging setting information 403 by the PC 200 or the like (instead of the digital camera 100).

[0092] exist Figure 4 In the example configuration of the image file shown, the annotation recording area A 404 and the annotation recording area B 405 are recorded in one metadata recording area 401. However, two metadata recording areas 401-1 and 401-2 may be provided, and the metadata recording area 401-1 may be an area for recording metadata using the digital camera 100 during image capture. The metadata recording area 401-2 may be an area for recording information generated by the PC 200 or the like based on the image data 402 and the image capture setting information 403 after image capture. In this case, the metadata recording area 401-1 includes the image capture setting information 403 and the annotation recording area A 404, and the metadata recording area 401-2 includes the annotation recording area B 405.

[0093] This embodiment provides a configuration for recording annotations recorded by a camera and annotations recorded by another device in different metadata recording areas, thereby distinguishing the annotations recorded by the camera. However, this configuration is not limiting. For example, information about the device that recorded the annotation can be assigned to each annotation to distinguish which device recorded the annotation.

[0094] Figure 5A and Figure 5B The following is a block diagram illustrating an example configuration of annotations according to an embodiment of the present invention. Each annotation includes annotation description information ("description") and annotation area information ("area"). The annotation description information is the string information of the annotation, while the annotation area information is the coordinate and size information indicating the area in the image.

[0095] Figure 5A An example configuration is shown which has no layer structure (nested structure) and stores annotation description information and annotation area information.

[0096] and Figure 5A The construction shown in Figure 5B An example construction with a layer structure and storing sub-annotations is shown below. Figure 5B In FIG, a layer structure having one layer is shown as an example, but the number of layers in the layer structure is not limited. Figure 5B In the example, each sub-annotation in sub-annotations S1 and S2 has annotation description information and annotation area information.

[0097] Regarding the configuration of annotations, whether each annotation can be recorded depends on, for example, the device that records the annotation and the standard for recording the annotation.

[0098] Figure 6A and Figure 6B They are shown respectively Figure 5A and Figure 5B A specific example of the annotation is shown in the figure.

[0099] exist Figure 6A In , the description of the annotation is “face” and the annotated area is represented by a rectangle.

[0100] exist Figure 6B In, with Figure 6A Similarly, the annotation is described as "face" and the annotated area is represented by a rectangle. In addition, annotations described as "eyes" and "nose" are stored in a layer structure.

[0101] In addition to rectangles, the annotated area can also be recorded in various shapes such as polygons and circles. Moreover, the information to be recorded as each annotation is not limited to the information in the above examples, and various related information such as the device that generated the annotation, the reliability of the annotation, and the recording date and time of the annotation can also be recorded.

[0102] First embodiment

[0103] Although the series of operations according to the first embodiment of the present invention can be performed by the digital camera 100 , the series of operations will be described as being performed by the PC 200 as an example.

[0104] Figure 7 is a diagram showing a processing unit included in the PC 200 according to the first embodiment of the present invention.

[0105] The control unit 201 (processing unit) of the PC 200 has Figure 3 In addition to the units shown, it also includes Figure 7 The control unit 201 of the PC 200 functions as a controller including the control unit 1000 to 1002 by loading a predetermined program stored in the ROM 202 into the work RAM and executing the program. Figure 3 and Figure 7 The processing unit of the unit shown is controlled by the control unit (processing unit) according to the program read by the control unit 201 (processing unit) Figure 7 The units shown and Figure 2 The units of the PC 200 shown are implemented Figure 8 When the digital camera 100 performs the operation according to the present embodiment, the system control unit 50A of the digital camera 100 includes Figure 7 The processing unit shown.

[0106] The annotation correlation determination unit 1000 determines whether multiple annotations are correlated based on the information contained in the annotations. In this embodiment, multiple annotations are determined to be correlated when the following conditions are met: the annotations have the same description; the areas indicated by the annotations overlap by 50% or more (the overlap is greater than or equal to a threshold or a predetermined value).

[0107] In this embodiment, the above two conditions are used. However, if the descriptions of the annotations do not include the same string information, it can be determined that the annotations are related. For example, language differences such as "human" in English and "hito" in Japanese (meaning "human"), expression differences such as "person" and "human", expression granularity differences such as "person" and "male", etc. can also be considered, and the similarity between the descriptions can be calculated to make a determination. For example, it is determined that the mouth, eyes, ears, etc. are related to the face. Multiple annotations with a layer structure (for example, see Figure 6B ) is related without using the description of the annotation. In addition, the threshold of region overlap can be changed according to the situation. For example, when the region size is less than the threshold, the annotation can be determined to be related when the region overlap is greater than or equal to 80%. When the region size is greater than the threshold, the annotation can be determined to be related when the region overlap is greater than or equal to 50%. In addition, the threshold for determining region overlap can be changed according to the description of the annotation. For example, when the region overlaps by 70%, annotations with the same description can be determined to be related, and for annotations with different descriptions, the threshold for overlap determination can be changed. The threshold for overlap determination can be changed according to the relevance of the description. For example, for the mouth, 80% of the proportion of the face is set as the threshold for overlap determination. For the eyes, 80% of the proportion of the face is set as the threshold for overlap determination; for the ears, 5% of the proportion of the face is set as the threshold for overlap determination. In addition, annotations whose descriptions are related but the regions may not overlap (for example, face and ears) may be determined to be related even if the regions do not overlap.

[0108] The annotation integration determination unit 1001 determines whether to integrate related annotations, for example, by using the result obtained by the annotation relevance determination unit 1000. In this embodiment, in addition to the condition that the annotation relevance determination unit 1000 determines that the annotations are relevant, it is determined that annotations are integrated when the following condition is met: the annotations are recorded in different recording areas.

[0109] When annotations are recorded in different recording areas, duplicate recording is more likely to occur than when annotations are recorded in the same recording area. Therefore, the above-mentioned conditions are set in this embodiment.

[0110] However, this condition is not essential, and annotations recorded in the same recording area may be determined to be integrated. Information recorded in an image other than the annotations may be used to determine the integration of the annotations. For example, in the case where the annotations have a layered structure, the integration of each "parent" annotation may be considered to determine the integration of the child annotations.

[0111] The annotation integration unit 1002 integrates the annotations determined to be integrated by the annotation integration determination unit 1001 .

[0112] In this embodiment, the annotations determined to be integrated by the annotation relevance determination unit 1000 are integrated as follows.

[0113] Annotate cases with the same description

[0114] The description of the integrated annotation is set as the description of the annotation to be integrated, and the area of ​​the integrated annotation is set to include all areas indicated by the area of ​​the annotation to be integrated.

[0115] Situations where the annotation descriptions are not identical but are determined to be related

[0116] When the annotations are in different languages, the description of the integrated annotation is set to the description of the annotation in the specific language. When the annotations have different expression granularities, the description of the integrated annotation is set to the annotation description with the larger granularity. For example, when it is determined that the face is related to the eyes, the description of the integrated annotation is set to "face". The region of the integrated annotation is set to include all areas indicated by the region of the annotation to be integrated.

[0117] When annotations have a layer structure

[0118] The description of the integrated annotation is set to the description of the annotation at the top (parent) level of the hierarchy, and the area of ​​the integrated annotation is set to the area of ​​the annotation at the top (parent) level of the hierarchy or to all areas indicated by the area including the annotation to be integrated.

[0119] In this embodiment, the region where the annotations are integrated includes all regions of the annotations to be integrated. However, the region where the annotations are integrated may include only the overlapping regions, and various methods may be used.

[0120] The annotations are thereby integrated, and a new annotation is generated. The annotation integration unit 1002 outputs the integrated annotation and the annotation determined not to be integrated as annotations of the image data to the annotation display control unit 305 .

[0121] Figure 8 is a flowchart showing an operational procedure of the annotation integration process according to the first embodiment.

[0122] Figure 9 and Figure 10 Show Figure 8Specific examples of annotations used in the description of the flowchart shown.

[0123] In step S1001 , the image reading unit 300 reads an image file.

[0124] In step S1002 , the comment acquisition unit 301 acquires the comment recorded in the image file read in step S1001 .

[0125] like Figure 9 Specifically, annotations (a) and (b) are acquired from the annotation recording area A 404 , and annotation (c) is acquired from the annotation recording area B 405 .

[0126] In step S1003 , the annotation relevance determination unit 1000 determines the relevance of the annotation acquired in step S1002 .

[0127] In obtaining Figure 9 In the case of the annotations shown, the annotation correlation determination unit 1000 determines that annotation (b) and annotation (c) having the same description (ie, "face") and overlapping regions are correlated.

[0128] In step S1004, the comment correlation determination unit 1000 determines whether there is a related comment in the determination result of step S1003, and branches the process. If there is a related comment, the process proceeds to step S1005, otherwise the process proceeds to step S1007.

[0129] exist Figure 9 In the illustrated comment recording area A 404 and comment recording area B 405 , it is determined that comment (b) and comment (c) are related, so the process proceeds to step S1005 .

[0130] In step S1005 , the annotation integration determination unit 1001 determines whether to integrate the annotations determined to be relevant, and branches the process. If the annotations are to be integrated, the process proceeds to step S1006 , otherwise the process proceeds to step S1007 .

[0131] exist Figure 9 In the illustrated annotation recording area A 404 and the annotation recording area B 405 , annotation (b) and annotation (c) are recorded in different annotation recording areas, and are therefore determined to be “integrated.” The process then proceeds to step S1006 .

[0132] In step S1006, the annotation integration unit 1002 integrates the annotations determined to be integrated and generates a new annotation. Then, the annotation integration unit 1002 outputs the annotations not determined to be integrated (or the annotations determined not to be integrated) and the new annotation generated by integration to the annotation display control unit 305.

[0133] exist Figure 9 In the example shown, annotations (b) and (c) are combined to generate Figure 10 Then, annotation (a) and annotation (bc) are output as annotations of the image data.

[0134] In step S1007, the annotation display control unit 305 displays the annotation on the display unit 206 of the PC 200. At this time, the image display control unit 304 is also used to display the image and the annotation on the display unit 206 at the same time, such as Figure 10 As shown. The annotation display control unit 305 can obtain information about the integrated annotations from the annotation integration unit 1002 and display the integrated annotations and non-integrated annotations in different display forms. For example, such annotations can be displayed with frames of different colors, frames of different thicknesses, different frame lines (such as dotted frames and double-line frames), etc.

[0135] The annotation assigning unit 303 may not display the annotation but may generate a new image file by replacing the image file acquired in step S1001 with the annotation output by the annotation integrating unit 1002. In this case, the generated image file is stored in a recording medium using the image storage unit 302.

[0136] By integrating and outputting (e.g., displaying or storing) related annotations through the above-described procedures, it is possible to improve the visibility of the annotations to be displayed and simplify the annotations to be stored in the image.

[0137] Second embodiment

[0138] Although a series of operations according to the second embodiment can be performed by the PC 200, the series of operations will be described as being performed by the digital camera 100. Since the configuration of the digital camera 100 is substantially the same as that of the first embodiment, description thereof will be omitted.

[0139] Figure 11 1 is a diagram showing a processing unit included in a digital camera 100 according to a second embodiment of the present invention. Figure 3 In addition to the units shown, it also includes Figure 11 Units 2000 to 2003 are shown. Figure 11 The units shown may be configured so that one unit performs the processing of multiple units.

[0140] The system control unit 50A of the digital camera 100 can function as a system including a computer system by loading a predetermined program or the like stored in the nonvolatile memory 51 into the work RAM and executing the program. Figure 11The system control unit 50A (processing unit) of the digital camera 100 controls the system control unit 50A (processing unit) according to the program read by the system control unit 50A (processing unit). Figure 11 The units shown and Figure 1 The units of the digital camera 100 shown are implemented Figures 14 to 16 The flowchart shown.

[0141] The external comment acquisition unit 2000 reads, from the external recording medium 91 , a comment recorded in an external file associated with the image file read by the image reading unit 300 .

[0142] The identification information acquisition unit 2001 acquires the subject identification information recorded in the imaging setting information 403 in the image file read by the image reading unit 300. The subject identification information includes, for example, depth information.

[0143] The comment inclusion relationship determination unit 2002 determines the inclusion relationship of comments based on the subject recognition information and the region information in the comments that meet a plurality of criteria.

[0144] The layer reconfiguration unit 2003 reconfigures the layer structure of the annotations based on the inclusion relationship determined by the annotation inclusion relationship determination unit 2002 .

[0145] Figure 12A is a diagram showing an example of annotations included in an image file. Figure 12B Is concerned Figure 12A The diagram of the layer structure of the annotations shown in . Figure 4 As shown, each metadata recording area (comment recording area A and B) that conforms to different standards stores comments. That is, comments of different recording formats are recorded in respective different recording areas. Figure 12A and Figure 12B The region information of each annotation is not shown and the description information of each annotation is shown, but each annotation actually includes description information and region information.

[0146] like Figure 12A As shown, in the image file, the annotation recording area A 2100 (corresponding to Figure 4 In the annotation recording area A404 shown in FIG. 2105 , annotations are recorded in a manner that conforms to the first standard. Figure 4 In the comment recording area B 405 shown, comments are recorded in a manner that conforms to the second standard.

[0147] exist Figure 12AIn the example shown, annotation recording area A2100 contains annotations in a hierarchical structure. The first layer, i.e., the top layer, contains annotation 2101 (description = "person"). The layer below annotation 2101 contains annotation 2102 (description = "face"). The layer below annotation 2102 contains annotation 2103 (description = "eye") and annotation 2104 (description = "eye").

[0148] exist Figure 12B , as shown by reference numeral 2109, the square connected to the circle (2100) corresponding to the annotation recording area A (first standard) and the square connected to the circle (2105) corresponding to the annotation recording area B (second standard) indicate the annotations recorded in the respective annotation recording areas. Annotations (2101, 2106, and 2107) directly connected to circles 2100 and 2105 are annotations in the first layer. Annotation 2102 connected from annotation 2101 in the first layer is an annotation in the second layer. Annotations 2103 and 2104 connected from annotation 2102 in the second layer are annotations in the third layer. Each annotation has descriptive information describing the content indicated by the annotation and area information indicating in which area of ​​the image the annotation is located. Here, the annotations recorded in the annotation recording area A that meet the first standard are annotations recorded by the digital camera 100. The annotations in the annotation recording area B (second standard) are annotations recorded by any annotation distribution device other than the digital camera 100.

[0149] 13A to 13F The following display example is shown: when the image display control unit 304 displays the image 2200 on the display unit 23, the image 2200 of the image file read by the image reading unit 300 is displayed so that Figure 12A and Figure 12B The annotation shown is superimposed on the image 2200 by the annotation display control unit 305. 13A to 13F In the figure, rectangles 2201 to 2204, 2205 and 2206 respectively indicate Figure 12A and Figure 12B The areas corresponding to the annotations 2101 to 2104, 2106, and 2107 are shown. Rectangle 2201 represents a box indicating the area of ​​annotation 2101. Rectangle 2202 represents a box indicating the area of ​​annotation 2102. Rectangles 2203 and 2204 represent boxes indicating the areas of annotation 2103 and annotation 2104, respectively. Rectangle 2205 represents a box indicating the area of ​​annotation 2106. Rectangle 2206 represents a box indicating the area of ​​annotation 2107. Although 13A to 13F An example in which a frame indicating an area of ​​an annotation is displayed as the annotation is shown, but a description of the annotation may be displayed together with the area.

[0150] Figure 13A 4 shows a display example showing the annotations 2100 recorded in the annotation recording area A 404. Figure 13A , as an example, the annotation display control unit 305 displays annotations in all layers recorded in the annotation recording area A. Therefore, areas 2201 to 2204 corresponding to the annotations 2101 to 2104, respectively, are displayed.

[0151] Figure 13B 4 shows a display example showing the annotation 2105 recorded in the annotation recording area B 405. Figure 13B , as an example, the annotation display control unit 305 displays annotations in all layers recorded in the annotation recording area B. Therefore, areas 2205 and 2206 corresponding to the annotation 2106 and the annotation 2107, respectively, are displayed.

[0152] When the annotation recording area A and the annotation recording area B display respective annotations, the annotations to be displayed can be switched in response to the user operating the operation unit 63. That is, the annotation display control unit 305 switches the annotation recording area (standard) to be displayed in accordance with the operation of switching the annotation recording area (standard) to be displayed. Figure 13A and Figure 13B Switches between the display modes shown.

[0153] exist Figure 13A In the example, all the annotations recorded in the annotation recording area A are superimposed on the image 2200 so that the user can centrally check the recorded annotations. However, the amount of information is large and the visibility of the image 2200 is low. Therefore, it is possible to use Figure 12A The annotation hierarchy shown is used to display the annotations layer by layer.

[0154] Figure 13C Show Figure 12A In the illustrated annotation, only an example of a region 2201 corresponding to the annotation 2101 in the first layer is displayed in a superimposed manner. Figure 13C The image 2200 shown has a Figure 13A Greater visibility in the Figure 13C , areas 2202, 2203, and 2204 corresponding to annotation 2102 (face) and annotations 2103 and 2104 (eyes) included in annotation 2101 are not displayed. Figure 13CAs shown, area 2201 is displayed in a double-line frame to indicate that annotations exist in a layer lower than the annotation 2101 corresponding to area 2201. In the present embodiment, an area is displayed in a double-line frame to indicate that annotations exist in a layer lower than the area. Such an area is displayed with a frame having a display form different from that of a frame of a normal area (such as color, shape, line type, etc.). When an operation is performed by the operation unit 63 to select an area (i.e., area 2201) indicating that annotations exist in a layer lower than the area, the annotation display control unit 305 controls to display annotations in a layer lower than the selected area 2201. Examples of the operation of selecting an area include touching the area and selecting the area using a cross key, etc. In Figure 13C In response to the operation of selecting area 2201, as shown in FIG. Figure 13D As shown, an area 2202 corresponding to the annotation 2102 in the layer immediately below the annotation 2101 corresponding to the area 2201 is displayed in an overlay manner on the image 2200. Since the annotations 2103 and 2104 exist in a layer below the annotation 2102, the frame indicating the area 2202 is displayed as a double-line frame. The frame indicating the area 2201 is, for example, Figure 13D The diagram is shown in a dotted frame to indicate that the area 2201 is in a higher layer. When the area in the higher layer displayed in the dotted frame is selected, the display of the area of ​​the annotation in the lower layer can be hidden, and the frame indicating the selected area 2201 can be displayed in a double-line frame. That is, the diagram can be presented again. Figure 13C The display shown. Figure 13D When the area 2202 displayed in a double-line frame is selected, as shown in FIG. Figure 13E As shown, areas 2203 and 2204 corresponding to the annotation 2103 and the annotation 2104 in the layer lower than the annotation 2102 corresponding to the selected area 2202 are displayed. Figure 13E In , since there are no annotations subordinate to annotations 2103 and 2104, areas 2203 and 2204 are displayed in single-line frames. Figure 13E In FIG, regions 2201 and 2202 corresponding to annotations 2101 and 2102 (i.e., ancestors (roots) of annotations 2103 and 2104) are displayed in dotted boxes. However, only regions corresponding to annotations (parents) in the upper layer of annotations 2103 and 2104 may be displayed. This provides higher visibility of the annotations. Alternatively, as Figure 13F As shown, the annotations 2100 (annotations 2101 to 2104) can be displayed so that each layer is indicated by a frame having a different display form to make the hierarchical relationship of the annotations recognizable. Whether to press the key can be switched by operating the operation unit 63 through menu settings or the like. Figure 13F or to display all annotations in the area as shown Figures 13C to 13E The way shown shows the annotations in some layers.

[0155] As described above, in this embodiment, annotations with a hierarchical structure can be displayed layer by layer. Figure 12A and Figure 12B As shown, comments recorded in the comment recording areas A and B in different formats are not always recorded in the same hierarchical structure.

[0156] Therefore, in this embodiment, the annotations recorded in different annotation recording areas of the image file are subjected to hierarchical structure reconstruction processing (layer reconstruction processing). Layer reconstruction enables all annotations recorded in different annotation recording areas in the image file to be collectively displayed in a hierarchical form.

[0157] Figure 14 1 is a flowchart showing the operation procedure of the layer reconstruction processing performed by the processing unit according to the second embodiment. The processing unit performs the following processing on the image file 400 to be subjected to the layer reconstruction processing. Figure 14 In the layer reconstruction process shown, the comment recording area A (404) is used as a reference, and the comments in the comment recording area B (405) are added to the comments in the comment recording area A (404) to perform layer reconstruction.

[0158] First, in step S2300 , the processing unit reads comments from the comment recording area A ( 404 ) by using the comment acquisition unit 301 , and temporarily saves the read comments in the memory 25 .

[0159] In step S2301, the annotation acquisition unit 301 reads annotations from the annotation recording area A (404), and the layer reconstruction unit 2003 generates the annotation list 1. The annotation list 1 is a hierarchical annotation list in which the annotations are arranged in order from the lowest to the highest. The list also includes the annotations in the parent layer.

[0160] In step S2302, the annotation acquisition unit 301 reads annotations from the annotation recording area B (405), which is different from the recording area from which the annotations were read in step S2301. The layer reconstruction unit 2003 then generates an annotation list 2 based on the read annotations. An annotation list 2 is a hierarchical list of annotations arranged in order from the lowest to the highest level. This list also includes annotations in the parent layer, and can also store an addition completion flag indicating whether the hierarchical structure reconstruction process has added annotations. The external annotation acquisition unit 2000 can also read annotations recorded in a separate file.

[0161] In step S2303 , the layer reconstruction unit 2003 sets the variable b to 1. The variable b manages the index number in the comment list 2 .

[0162] In step S2304, the layer reconstruction unit 2003 compares the length of annotation list 2 with the variable b and checks whether all elements of annotation list 2 have been processed. If all elements of annotation list 2 have been processed, the process proceeds to step S2314. If not all elements have been processed, the process proceeds to step S2305.

[0163] In step S2305 , the layer reconstruction unit 2003 acquires the b-th annotation in the annotation list 2 as annotation B.

[0164] In step S2306 , the layer reconstruction unit 2003 sets the variable a to 1. The variable a manages the index number in the comment list 1 .

[0165] In step S2307, the layer reconstruction unit 2003 compares the length of annotation list 1 with the variable a and checks whether all elements of annotation list 1 have been processed. If all elements of annotation list 1 have been processed, the process proceeds to step S2313. If not all elements have been processed, the process proceeds to step S2308.

[0166] In step S2308 , the layer reconstruction unit 2003 obtains the ath annotation in the annotation list 1 as annotation A.

[0167] In step S2309, the annotation inclusion relationship determination unit 2002 determines whether annotation A and annotation B have an inclusion relationship. Figure 15 The annotation inclusion relationship determination process is described in detail.

[0168] If the annotation inclusion relationship determination unit 2002 determines in step S2310 that annotation A has an inclusion relationship with annotation B, the process advances to step S2311. If it is determined that annotation A does not have an inclusion relationship with annotation B, the process advances to step S2312.

[0169] In step S2311, the layer reconstruction unit 2003 performs an adding process for adding the annotation B determined to have an inclusion relationship with the annotation A to the annotation temporarily stored in the memory 25. This will be referred to hereinafter. Figure 16 This layer information allocation process is described in detail.

[0170] In step S2312, the layer reconstruction unit 2003 increments the variable a by 1, and the process returns to step S2307.

[0171] In step S2313 , the layer reconstruction unit 2003 increments the variable b by 1, and the process returns to step S2304 .

[0172] Through the processing of steps S2301 to S2313, the annotation inclusion relationship determination processing (S2309) is performed on all annotation combinations in the annotation recording area A (404) and the annotation recording area B (405). Then, if it is determined that the annotation combination has an inclusion relationship, the addition processing (S2311) is performed.

[0173] In step S2314, the annotation display control unit 305 displays the annotation on the image in a superimposed manner based on the annotation temporarily stored in the memory 25. That is, the annotation is displayed based on the annotation obtained by adding the annotation in the annotation recording area B (405) to the annotation in the annotation recording area A (404) in consideration of the hierarchical relationship. The display method of the annotation is as follows, as shown in FIG. Figures 13C to 13F As described above, in response to the operation on the operating unit 63, the annotations are displayed layer by layer.

[0174] In step S2315, the image storage unit 302 performs annotation storage processing for storing the annotation temporarily stored in the memory 25 in the image file 400. In the annotation storage processing, more specifically, the annotation obtained by adding the annotation in the annotation recording area B (405) to the annotation in the annotation recording area A (404) while taking into account the hierarchical relationship is recorded in the image file. The obtained annotation is different from the annotation assigned by the digital camera 100 when shooting, and is therefore recorded in a new annotation recording area in the image file. Alternatively, the obtained annotation can be recorded in the annotation recording area B.

[0175] Figure 15 2309 is a flowchart showing the operation procedure of the comment inclusion relationship determination process performed by the comment inclusion relationship determination unit 2002 in step S2309.

[0176] In step S2320, the annotation inclusion relationship determination unit 2002 determines whether area A of annotation A and area B of annotation B have an inclusion relationship based on information about the positions and sizes of areas A and B. If area A is larger than area B and area B is contained within area A, the process proceeds to step S2321. If area B is larger than area A and area A is contained within area B, the process proceeds to step S2322. If neither condition is met and area A and area B do not have an inclusion relationship, the process proceeds to step S2325. In determining an inclusion relationship, one area does not have to be completely contained within the other area, but can be contained with a margin. The amount of margin can vary depending on the description of the annotation.

[0177] In step S2321, the annotation inclusion relationship determination unit 2002 determines whether description A of annotation A is related to description B of annotation B such that description A includes description B. For example, when description A is "face" and description B is "nose," description A is determined to include description B. If it is determined that description A includes description B, the process proceeds to step S2323. If it is determined that description A does not include description B, the process proceeds to step S2325.

[0178] In step S2323, the annotation inclusion relationship determination unit 2002 determines whether description A of annotation A is the same as description B of annotation B. For example, descriptions in different languages, such as "human" in English and "hito" (meaning "human") in Japanese, are determined to be the same, and descriptions in different expressions, such as "person" and "human" are also determined to be the same. If description A is determined to be the same as description B, there are annotations with the same description, and annotation B does not need to be added. Therefore, the process enters Figure 14 , and processing proceeds to step S2313 of the previous step, and the next annotation in annotation list 2 is processed. If it is determined that description A is not identical to description B, the process proceeds to step S2324. If it is determined in step S2323 that annotation A and annotation B have the same description, the region information of annotation A and the region information of annotation B are integrated, as in the first embodiment. If the region information of annotation A and the region information of annotation B are to be integrated, the region information of annotation A temporarily stored in memory 25 is updated using the region information obtained as a result of the integration.

[0179] In step S2324, the annotation inclusion relationship determination unit 2002 determines that there is an inclusion relationship, and sets the variable Z of the method for annotation addition processing to 1. Then, the annotation inclusion relationship determination processing ends. Then, the processing enters Figure 14 . In step S2325, the annotation inclusion relationship determination unit 2002 determines that there is no inclusion relationship. Then, the annotation inclusion relationship determination processing ends. Then, the processing enters step S2310. In step S2322, the annotation inclusion relationship determination unit 2002 determines whether description B of annotation B contains description A of annotation A. For example, when description B is "face" and description A is "nose", it is determined that description B contains description A. If it is determined that description B contains description A, the processing enters step S2326. If it is determined that description B does not contain description A, the processing enters step S2325.

[0180] In step S2326, as in step S2323, the annotation inclusion relationship determination unit 2002 determines whether Description A of annotation A is identical to Description B of annotation B. If Description A is determined to be identical to Description B, a annotation with the same description exists, and there is no need to add annotation B. Therefore, processing proceeds to step S2313 and moves on to processing the next annotation in annotation list 2. If Description A is determined to be different from Description B, processing proceeds to step S2327. If it is determined in step S2326 that annotation A and annotation B have the same description, then, as in the first embodiment, the region information of annotation A and the region information of annotation B can be integrated. If the region information of annotation A and the region information of annotation B are to be integrated, the region information of annotation A temporarily stored in memory 25 is updated using the region information obtained as a result of the integration.

[0181] In step S2327 , the comment inclusion relationship determination unit 2002 determines whether comment A has a parent comment (a comment in an upper layer of comment A). If comment A has a parent comment, the process advances to step S2328 , otherwise the process advances to step S2330 .

[0182] In step S2328 , the comment inclusion relationship determination unit 2002 acquires the parent comment of comment A (the comment in the upper layer of comment A) as comment C.

[0183] In step S2329, the annotation inclusion relationship determination unit 2002 determines whether area B of annotation B and area C of annotation C have an inclusion relationship based on information about the positions and sizes of area B and area C. The method for determining the inclusion relationship is similar to the method in step S2320. If area B is larger than area C and area C is included in area B, the process proceeds to step S2325, otherwise the process proceeds to step S2330. If area B is larger than area C and area C is included in area B, it is expected that annotation B is added to a higher layer, not to the upper layer of annotation A. Therefore, in step S2325, although there is actually an inclusion relationship, it is determined that there is no inclusion relationship, and the process proceeds to the next process, so that in the inclusion relationship determination process, it is determined that there is an inclusion relationship with the annotation in the layer higher than annotation A, and the addition process is performed.

[0184] In step S2330, the annotation inclusion relationship determination unit 2002 determines that an inclusion relationship exists and sets the variable Z of the method for annotation adding processing to 2. Then, the annotation inclusion relationship determination processing ends. Then, the processing proceeds to step S2310.

[0185] Figure 16 2311 is a flowchart showing the operation procedure of the adding process performed by the layer reconstruction unit 2003 in step S2311.

[0186] In step S2340 , the layer reconstruction unit 2003 determines the variable Z. If the variable Z is 1, the process proceeds to step S2341 . If the variable Z is 2, the process proceeds to step S2342 .

[0187] In step S2341, the layer reconstruction unit 2003 adds the annotation B to the next layer of the annotation A among the annotations temporarily stored in the memory 25. The addition completion flag is set for the added annotation B.

[0188] In step S2342, the layer reconstruction unit 2003 determines whether the addition completion flag has been set for annotation B. If the addition completion flag has been set, that is, if it is determined that annotation B has been added to the memory 25, the process proceeds to step S2343. If it is determined that annotation B has not been added to the memory 25, the process proceeds to step S2344.

[0189] In step S2343, the layer reconstruction unit 2003 moves annotation A to the next layer of annotation B among the annotations temporarily stored in the memory 25. If annotation B is an annotation added as an ancestor with reference to the ancestor addition flag described below, the processing of step S2343 has already been performed. Therefore, the processing of step S2343 is not performed.

[0190] In step S2344 , the layer reconstruction unit 2003 determines whether annotation A has a parent annotation C (in an upper layer of annotation A). If annotation A has a parent annotation, the process advances to step S2346 , otherwise the process advances to step S2345 .

[0191] In step S2345, the layer reconstruction unit 2003 adds annotation B to the upper layer of annotation A among the annotations temporarily stored in the memory 25. In addition, if annotation B has a parent annotation, the ancestor annotation of annotation B is added to a layer higher than annotation A while maintaining the hierarchical structure. An ancestor annotation is an annotation that traces back to the parent of the root (for example, a parent, the parent of the parent, the parent of the parent of the parent) and does not include the annotation of the child of the parent of the parent. An addition completion flag is set for the added annotation B and its ancestor annotation. An ancestor addition flag is set for the annotation added as an ancestor, and the ancestor addition flag is used in step S2343.

[0192] In step S2346, the layer reconstruction unit 2003 adds a layer between annotations A and C among the annotations temporarily stored in the memory 25, and adds annotation B to the added layer. Then, the addition completion flag is set for the added annotation B.

[0193] As described above, adding processing is performed to add the comment to the comments temporarily saved in the memory 25, and then the process advances to step S2313 and advances to processing of the next comment in the comment list 2.

[0194] As described above, in this embodiment, layer reconstruction processing is performed based on the inclusion relationship of the annotation area and the correlation of the annotation description. However, this configuration is not restrictive, and in addition to the inclusion relationship of the annotation area and the correlation of the annotation description, the inclusion relationship of the annotations can also be determined using, for example, the depth information acquired by the identification information acquisition unit 2001. Even when the annotation area and description are used to determine that the annotations have an inclusion relationship, if the depth information of the subject indicated by the annotations is significantly different, it can be determined that the annotations do not have an inclusion relationship.

[0195] 17A to 17D It is shown that Figure 12A and Figure 12B The annotations shown are obtained by performing layer reconstruction processing.

[0196] First, in step S2300, the annotations in the annotation recording area A (2100) are temporarily recorded in the memory 25. Therefore, Figure 17A As shown by reference numeral 2400 in FIG, annotations 2101 to 2104 are temporarily recorded in the memory 25 while maintaining the hierarchical structure. Thereafter, based on the information about the lower layer, the inclusion relationship determination processing in step S2309 is performed on each of annotations 2106 and 2107 in the annotation recording area B (2105) written in the annotation list 2. Annotation 2106 and annotation 2107 are located in the same layer, so either one of them can be processed first. Here, annotation 2106 is processed first. It is determined that annotation 2106 does not have an inclusion relationship with annotations 2103 and 2104. Thereafter, processing for determining the inclusion relationship with annotation 2102 in the upper layer of annotations 2103 and 2104 is performed (S2309), and it is determined that annotation 2106 has an inclusion relationship with annotation 2102, and variable Z is 1. Then, in the adding process of step S2311, the annotation 2106 is added to the next layer of the annotation 2102 in the process of step S2341. Figure 17B The annotations 2401 to 2405 shown are temporarily recorded in the memory 25. Thereafter, the annotation 2107 is processed in a similar manner, and the annotation 2107 (i.e., the annotation 2406) is added to the next layer of the annotation 2102, as shown in FIG. Figure 17C As described above, the layer reconstruction process generates Figure 17C Notes shown.

[0197] Figure 17D Show Figure 17C The annotation hierarchy is shown. When using annotation 2100 as a reference to Figure 12A and Figure 12B When the annotations 2100 and 2105 shown in FIG. 1 are subjected to layer reconstruction processing, a layer having Figure 17DThe layer structure 2700 shown Figure 17C Notes shown.

[0198] In this embodiment, the annotations in the annotation record area B (405) are added to the annotations in the annotation record area A (404) in consideration of the hierarchical relationship. Alternatively, the layer reconstruction process may be performed using the annotation record area B (405) as a reference so that the annotations in the annotation record area A (404) are added to the annotations in the annotation record area B (405). In this case, the layer reconstruction process is performed by replacing the annotation record area A and the annotation record area B. Figures 14 to 16 The processing shown.

[0199] Will refer to 18A to 18D To describe when the annotation in the annotation recording area A is added to the annotation in the annotation recording area B as a reference, Figure 12A and Figure 12B Layer reconstruction processing performed based on the annotations shown in .

[0200] First, in step S2300, the annotations in the annotation recording area B (2105) are temporarily recorded in the memory 25. Therefore, Figure 18A As shown by reference numeral 2500 in FIG, annotations 2106 and 2107 are temporarily recorded in the memory 25 while maintaining the hierarchical structure. Thereafter, based on the information about the lower layer, the inclusion relationship determination processing in step S2309 is performed on annotations 2101 to 2104 in the annotation recording area A (2100) written in the annotation list 1. Annotation 2103 and annotation 2104 are located in the same layer, so either one can be processed first. Since it is determined that annotations 2103 and 2104 do not have an inclusion relationship with annotations 2106 and 2107, annotations 2103 and 2104 are not added to the memory 25. Next, in the inclusion relationship determination processing (S2309), it is determined that annotation 2102 has an inclusion relationship with annotation 2106, and the variable Z is 2. Then, in the adding process (S2311), it is determined that the annotation 2102 has not been added ("No" in step S2342), and it is determined that the annotation 2106 has no parent annotation ("No" in step S2344). Therefore, in step S2345, as shown in FIG. Figure 18BAs shown, annotation 2102 (i.e., annotation 2502) and annotation 2101 (i.e., annotation 2501), which is the parent (ancestor) of annotation 2102, are added to a layer higher than annotation 2106 while maintaining the hierarchical structure. At this time, annotation 2101 and annotation 2102 are assigned an addition completion flag, and annotation 2101 is further assigned an ancestor addition flag. Then, the addition completion flag is set for annotation 2101 and annotation 2102. Thereafter, in the inclusion relationship determination processing (S2309), it is determined that annotation 2102 has an inclusion relationship with annotation 2107, and variable Z is 2. Then, in the addition processing (S2311), it is determined that annotation 2102 has been added ("Yes" in step S2342). Then, in step S2343, as shown Figure 18C As shown, the annotation 2107 is moved to the next layer of the annotation 2102. Thereafter, the inclusion relationship determination process (S2309) is performed on the annotation 2101. Since the ancestor addition flag is set in the annotation 2101, the addition process (S2311) is not performed on the annotation 2101. In the inclusion relationship determination process (S2309), it can be determined that the annotation with the ancestor addition flag set has no inclusion relationship. As described above, the layer is generated by the layer reconstruction process. Figure 18C Notes shown.

[0201] Figure 18D Show Figure 18C The annotation hierarchy is shown. When using annotation 2105 as a reference to Figure 12A and Figure 12B When the annotations 2100 and 2105 shown in FIG. 1 are subjected to layer reconstruction processing, a layer having Figure 18D The layer structure 2700 shown Figure 18C Notes shown.

[0202] Since the result of layer reconstruction processing varies depending on which annotation record area is used as a reference, the user can be allowed to select which annotation record area to use as a reference. Alternatively, the record area in which annotations are used as a reference can be determined based on the number of layers. For example, layer reconstruction processing can be performed using annotations with a large number of layers as a reference.

[0203] Figure 19 The following is shown based on the Figure 17C The annotations shown are examples of annotations displayed in boxes that differ in display form for each layer. 13A to 13F In the present invention, only the layers for each recording area are considered for displaying the annotations. However, the layer reconstruction process enables the annotations in a plurality of recording areas to be collectively displayed in a hierarchical manner.

[0204] In the above description, for example, annotations with a hierarchical structure are recorded in the annotation record area A, and annotations without a hierarchical structure are recorded in the annotation record area B. However, in the case where annotations with a hierarchical structure are recorded in the annotation record areas A and B, layer reconstruction processing can also be performed. Alternatively, layer reconstruction processing can be performed on annotations in three or more recording areas instead of on annotations in two recording areas. The above-mentioned layer reconstruction processing is an example of processing for reconstructing the hierarchical structure of multiple annotations, and the hierarchical structure can be reconstructed in any other process. In addition, layer reconstruction processing can be performed after the processing of integrating annotations according to the first embodiment is performed. In the above-mentioned embodiment, layer reconstruction processing is performed on annotations in multiple recording areas. However, as in the present embodiment, annotations recorded in one recording area and without a hierarchical structure can be organized in a hierarchical structure based on the inclusion relationship of the annotation areas and the relevance of the annotation descriptions. Alternatively, after the annotations with an inclusion relationship are layered and organized in a hierarchical structure based on the inclusion relationship of the annotation areas and the relevance of the annotation descriptions, layer reconstruction processing according to the present embodiment can be performed on the annotations in each recording area.

[0205] Third embodiment

[0206] Although a series of operations according to the third embodiment of the present invention can be performed by the PC 200, the series of operations will be described as being performed using the digital camera 100. Since the configuration of the digital camera 100 is substantially the same as that in the first embodiment, its description will be omitted. The system control unit 50A (processing unit) of the digital camera 100 controls the operation according to a program read by the system control unit 50A (processing unit). Figure 3 The units shown and Figure 1 The units of the digital camera 100 shown are implemented Figures 20 to 22 The flowchart shown.

[0207] 23A to 23D 、 Figure 24A and Figure 24B Show Figures 20 to 22 Specific examples of annotations used in the description of the flowchart shown in .

[0208] Figure 20 is a flowchart showing the operation procedure of the image display process according to the third embodiment.

[0209] In step S3001 , the image reading unit 300 reads an image file to be displayed.

[0210] In step S3002 , the comment acquisition unit 301 acquires the comment of the image file read in step S3001 .

[0211] In step S3003, the image display control unit 304 displays the image, and the annotation display control unit 305 causes the display unit 23 to display the annotation acquired in step S3002. Then, the processing ends. Figure 21 The processing of step S3003 is described in detail.

[0212] Figure 21 Show Figure 20 In the annotation display process, each annotation acquired by the annotation acquisition unit 301 is processed. Figure 21 The processing shown.

[0213] In step S3101, the annotation display control unit 305 determines whether the annotation to be processed is an annotation recorded by the digital camera 100. As described above, according to this embodiment, annotations are recorded in the annotation recording area A 404 when the annotation is recorded during image capture using the digital camera 100, and in the annotation recording area B 405 in other cases. Therefore, if the annotation to be processed is information recorded in the annotation recording area A 404, the annotation display control unit 305 determines that the annotation to be processed is an annotation recorded by the digital camera 100. If the annotation to be processed is an annotation recorded by the digital camera 100, the process proceeds to step S3102. If the annotation to be processed is not an annotation recorded by the digital camera 100, or is not an annotation recorded by a device other than the digital camera 100 (e.g., a PC), the process proceeds to step S3104.

[0214] In step S3102, the annotation display control unit 305 determines whether the annotation determined in step S3101 is interpretable. If the annotation is interpretable, the process proceeds to step S3103. If the annotation is not interpretable, the process proceeds to step S3105. For example, whether the annotation is interpretable can be determined based on whether the annotation is information recorded by the digital camera 100 in accordance with the Exif standard. Information about the imaging device included in the imaging setting information is used to determine whether the annotation has been recorded by the digital camera 100. In the case where the annotation is an annotation recorded in accordance with the Exif standard, whether the annotation is interpretable can be determined based on the version of the Exif standard. Even in the case where the annotation is an annotation recorded by another camera, if the annotation is information recorded by a camera of the same manufacturer as the digital camera 100 or if character string information that can be analyzed is recorded, the annotation display control unit 305 can determine that the annotation is interpretable.

[0215] In step S3103, Figure 23AAs shown, the annotation display control unit 305 controls to display a frame and character string information in an overlay manner on the image based on the annotation. That is, the annotation display control unit 305 controls so that the area indicated by the annotation area is displayed in a frame, and the character string describing the annotation is displayed near the frame. Then, the processing of the annotation to be processed ends. Figure 23A When the operation unit 63 of the digital camera 100 is operated in the manner shown to display an enlarged image on the display unit 23, the annotation display control unit 305 adjusts the display position of the frame according to the enlarged image and displays the frame as shown in FIG. Figure 24A In addition, only annotations whose entire area is contained within the zoom range are displayed. Figure 24A Show that Figure 23A In the enlarged image shown, the annotation "Eye" is displayed because its entire area is within the display range. In contrast, the annotations "Mouth" and "John" are displayed without a frame or text string because only parts of their areas are within the display range.

[0216] In step S3104, the annotation display control unit 305 determines whether the area information of the annotation to be processed is analyzable. For example, it determines whether the annotation recorded in the annotation recording area A 404 is based on a standard that can be analyzed by the digital camera 100. If the area information of the annotation to be processed is analyzable, the process proceeds to step S3105; otherwise, the process proceeds to step S3106.

[0217] In step S3105, Figure 23B As shown, the annotation display control unit 305 controls to display only the frame indicated by the annotation area in a superimposed manner on the image without displaying text information. Then, the processing of the annotation to be processed ends. In the case where the image is displayed as an enlarged view, as in step S3103, the display position of the annotation frame is adjusted, and the annotation to be displayed is adjusted (only the annotations whose entire area is included in the display range are displayed). Therefore, as the image is displayed as an enlarged view, Figure 23B The result of the image shown is obtained Figure 24B The image shown.

[0218] In step S3106, no annotation is recorded in the image. Figure 23C As shown, the image display control unit 304 displays the image on the display unit 23, and the annotation display control unit 305 does not display the annotation. Then, the processing for the annotation to be processed ends.

[0219] and Figure 21As in the example, the annotation display control unit 305 determines how each of the plurality of annotations will be displayed. Then, based on the determination result, display data of the plurality of annotations may be generated at the same time and the annotations may be displayed on the display unit 23.

[0220] 23A to 23D A display example is shown when three annotations indicating “John,” “eyes,” and “mouth” are recorded. Figure 23A A display example is shown when the process of S3103 is performed on all annotations. Figure 23B A display example is shown when the process of S3105 is performed on all annotations. Figure 23C Shows an example of display when all annotations are processed in S3106. Figure 23A In the case where annotations are recorded in a plurality of different annotation recording areas (e.g., annotation recording area A 404 and annotation recording area B 405), the determination result varies depending on the annotation recording area. Therefore, for example, a display such as Figure 23D Although all annotated data are analyzable, Figure 23D A display example is shown when a comment indicating “John” is recorded in the comment recording area B 405 and comments indicating “mouth” and “eyes” are recorded in the comment recording area A 404 .

[0221] Figure 22 A modified example of image display processing is shown. Figure 22 In the case of continuously playing back and displaying images at high speed during playback using the digital camera 100, annotations are provided with the reference image. Figure 21 Descriptions are displayed in a different way than normal comments.

[0222] Since the processing of step S3201 and step S3202 is the same as Figure 20 The processing of step S3001 and step S3002 is similar, so their description will be omitted.

[0223] In step S3203, the image display control unit 304 determines whether the image to be displayed on the display unit 23 is undergoing a high-speed image feed operation performed by the operation unit 63 of the digital camera 100. Then, if it is determined that the image feed operation is being performed at high speed, the process proceeds to step S3204. If it is determined that the image feed operation is not being performed at high speed, the process proceeds to step S3208.

[0224] In step S3204, the annotation display control unit 305 determines whether the annotation acquired in step S3202 includes an annotation recorded by the digital camera 100. A method similar to the method in step S3101 is used to determine whether the annotation includes an annotation recorded by the digital camera 100. If it is determined that the annotation includes an annotation recorded by the digital camera 100, the process advances to step S3205; otherwise, the process advances to step S3207.

[0225] In step S3205, the annotation display control unit 305 determines whether the annotation recorded by the digital camera 100 is interpretable. This determination is made by determining whether the digital camera 100 has recorded an annotation based on information about the imaging device included in the imaging setup information, as in step S3102, or by determining whether the annotation is interpretable based on the version of the Exif standard. This process avoids the time-consuming determination operation, enabling high-speed playback. If the annotation is determined to be interpretable, the process proceeds to step S3206; otherwise, the process proceeds to step S3207.

[0226] In step S3206, the image display control unit 304 causes the display unit 23 to display the image file read in step S3201. The annotation display control unit 305 then controls the display of a frame indicating the area of ​​the annotation recorded by the digital camera 100 and text information describing the annotation. In this embodiment, when there are multiple annotations, multiple frames and text information are displayed based on the multiple annotations. However, only the frames and text information of the annotations in the top layer may be displayed. Alternatively, multiple frames may be displayed without text information.

[0227] In step S3207, the image display control unit 304 causes the display unit 23 to display the image of the image file read in step S3201. Then, the annotation display control unit 305 controls not to display the annotation.

[0228] In step S3208, as in step S3003, the image display control unit 304 and the annotation display control unit 305 refer to Figure 21 Normal comment display processing for descriptions.

[0229] As described above, in the modified example, annotations recorded by the digital camera 100 are displayed even during high-speed playback because such information can be easily processed by the digital camera 100, while annotations not recorded by the digital camera 100 are not displayed during high-speed playback because such information requires time to process and analyze. Through the above-described procedure, the display content of the annotations recorded in the image is switched, that is, whether to display the annotations recorded in the image is determined based on the recording area in which the annotations are recorded, etc. In the case where similar annotations are included in different recording areas, the annotations recorded in one of the recording areas are displayed in a simple manner. This can improve the visibility when the annotations are displayed. Annotations that display only a frame and annotations that display both a frame and text information are displayed with different frame display forms (e.g., color, shape, or line type such as dotted line and double line).

[0230] Fourth embodiment

[0231] The fourth embodiment describes a process for sorting images to which focus levels (hereinafter referred to as "focus levels") as annotations are assigned in an overlapping manner and displaying the images in a list view. This embodiment describes an example of image sorting processing using focus level values. However, for example, any other processing such as image filtering processing or image search processing may be performed. In addition, for example, the calories of food in an image, the age and height of a person in an image, etc. may be processed as long as the value can be interpreted numerically like the focus level value. Hereinafter, descriptions of points that are the same as those in the first to third embodiments (for example, the same configuration and operation) will be omitted as appropriate.

[0232] Although a series of operations according to the present embodiment can be performed by the digital camera 100 , the series of operations will be described as being performed by the PC 200 .

[0233] Figure 25A and Figure 25B is a diagram showing an image to which overlapping focus level values ​​are assigned as annotations. Figure 25A and Figure 25B 1 shows an example in which the digital camera 100 assigns a focus level value 4001 as an annotation during image capture, and an image processing application or the like assigns a focus level value 4002 as an annotation. Figure 25A In the example shown, the digital camera 100 assigns a focus area 4003 as autofocus information when capturing an image. Figure 25B In the illustrated example, an image processing application or the like assigns an eye region 4004 and a mouth region 4005 as annotations.

[0234] Figure 26 2 is a block diagram showing a processing unit included in a PC 200 according to a fourth embodiment of the present invention. In the fourth embodiment, the control unit 201 (processing unit) of the PC 200 includes Figure 7 In addition to the units shown, it also includes Figure 26 The control unit 201 of the PC 200 functions as a computer including a computer 200 by loading a predetermined program stored in the ROM 202 into the work RAM and executing the program. Figure 26 The processing unit of the unit shown is controlled by the control unit 201 (processing unit) according to the program read by the control unit 201 (processing unit) Figure 26 The units shown and Figure 2 The units of the PC 200 shown are implemented Figure 27 and Figure 28 The annotation correlation determination unit 1000 determines whether there are overlapping annotations.

[0235] The annotation interpretation unit 4100 determines whether the acquired annotation can be interpreted as a numerical value. In this embodiment, it is analyzed whether the annotation is in a suitable format that can be used as a keyword for performing image sorting processing.

[0236] The subject recognition information acquisition unit 4101 acquires subject recognition information recorded in an image by the digital camera 100 when shooting, such as autofocus information, a depth map, and sensor information such as global positioning system (GPS) information. Figure 25A The focus area 4003 shown indicates the auto focus information.

[0237] The reliability determination unit 4102 determines the reliability of the annotation using the reliability assigned to the annotation itself, the subject identification information acquired by the subject identification information acquisition unit 4101 , the detail level of the annotation, the type of the annotation, and the like.

[0238] In the present embodiment, in the case where reliability is assigned to the target annotation, the reliability determination unit 4102 determines the reliability using the reliability.

[0239] Furthermore, in the present embodiment, in the case where subject identification information is recorded in the target annotation, the reliability determination unit 4102 determines reliability using the subject identification information.

[0240] For example, the target annotation is assigned by the digital camera 100 Figure 25A Note 4001 shown in . In contrast, in being able to obtain Figure 25AIn the case of the autofocus information indicated in the focus area 4003 shown in FIG4 (which is recorded as subject recognition information by the digital camera 100), if the annotation 4001 and the focus area 4003 are close to each other, the reliability determination unit 4102 determines that the reliability is high. Although the autofocus information has been described as an example of subject recognition information, any other information such as a depth map can be used for determination.

[0241] Furthermore, in this embodiment, the level of detail of the annotation is determined based on whether an annotation more detailed than the target annotation is assigned. Figure 25B In the case where the target annotation is assigned to a person's face as in the illustrated annotation 4002 , if more detailed regions, namely, the eye region 4004 and the mouth region 4005 are assigned, the reliability determination unit 4102 determines that the reliability is higher.

[0242] Moreover, in this embodiment, the type of annotation refers to the standard based on which the annotation is recorded or the person, device, etc. who recorded the annotation. For example, the reliability determination unit 4102 may determine that an annotation recorded by a specific person or a specific device has high reliability.

[0243] Furthermore, in this embodiment, if reliability cannot be determined even using the above information, a new annotation is generated from the overlapping annotations, and the reliability of the generated annotation is determined to be high. For example, as in this embodiment, when the focus level is recorded as a numerical value as an overlapping annotation of the image, a weighted average of the overlapping focus level values ​​is calculated to generate an annotation, and the generated annotation is treated as a high-reliability annotation.

[0244] The annotation reference unit 4103 selects annotations for image sorting processing among the overlapping annotations based on the reliabilities determined by the reliability determination unit 4102. For example, annotations determined to have high reliability by the reliability determination unit 4102 are selected as sort keys for image sorting processing.

[0245] Figure 27 and Figure 28 1 is a flowchart showing an operation procedure of the image sorting process according to the fourth embodiment. In the image sorting process according to the fourth embodiment, before performing the sorting process of images, a focus level value used as a keyword for the sorting process is determined for each image.

[0246] In step S4201 , the image file to be sorted is read using the image reading unit 300 .

[0247] In step S4202 , the annotation acquisition unit 301 is used to acquire an annotation from the image read in step S4201 .

[0248] In step S4203, it is determined whether the annotations acquired in step S4202 include overlapping annotations using the annotation correlation determination unit 1000. If the annotations include overlapping annotations, the process proceeds to step S4204. If the annotations do not include overlapping annotations, the process proceeds to step S4211.

[0249] In step S4204, the annotation interpretation unit 4100 analyzes the focus level value based on the overlap annotation determined in step S4203.

[0250] In step S4205, the control unit 201 determines whether to perform the sorting process in simple inspection mode or detailed inspection mode when sorting the images. Simple inspection mode is a mode in which the sorting process is performed by referring to any one of the overlapping focus level values. Detailed inspection mode is a mode in which the sorting process is performed using multiple focus level values ​​among the overlapping focus level values. It is assumed that the user operating the image processing apparatus according to this embodiment has previously selected simple inspection mode or detailed inspection mode before issuing an instruction to start the sorting process. It is also assumed that, when selecting simple inspection mode, the user has previously selected which focus level value to prioritize among the overlapping focus level values. If the user has selected simple inspection mode, the process proceeds to step S4206. If the user has selected detailed inspection mode, the process proceeds to step S4207.

[0251] In step S4206, the control unit 201 refers to the focus level value selected by the user in step S4205 among the overlapping focus level values ​​as a key for sorting processing.

[0252] In step S4207, the control unit 201 determines whether the overlapping focus level values ​​match. If the overlapping focus level values ​​match, the process proceeds to step S4208. If the overlapping focus level values ​​do not match, the process proceeds to step S4209.

[0253] In step S4208, the control unit 201 refers to any one of the matched focus level values ​​among the overlapping focus level values ​​as a key for the sorting process.

[0254] In step S4209, the reliability determining unit 4102 refers to a focus level value with higher reliability among the overlapping focus level values ​​as a key for sorting processing. A method for determining reliability will be described with reference to steps S4213 to S4223 described below.

[0255] In step S4211 , the focus level value is analyzed based on the annotation determined in step S4203 using the annotation interpretation unit 4100 .

[0256] In step S4212, the control unit 201 refers to the focus level value analyzed in step S4211 as a key for sorting processing.

[0257] In step S4213, the reliability determination unit 4102 is used to determine whether reliability is recorded in each overlapping annotation. If reliability is recorded in the annotation itself, the process proceeds to step S4214. If reliability is not recorded in the annotation itself, the process proceeds to step S4215.

[0258] In step S4214, the reliability recorded in the annotation itself among the overlapping annotations is referenced by the reliability determination unit 4102. The focus level value written in the annotation with high reliability is referenced as a key for the sorting process.

[0259] In step S4215, the reliability determination unit 4102 determines whether an annotation assigned with autofocus information is included in the overlapping annotations. If an annotation assigned with autofocus information exists, the process proceeds to step S4216. If an annotation assigned with autofocus information does not exist, the process proceeds to step S4218.

[0260] In step S4216, the reliability determination unit 4102 determines whether the area indicated by the autofocus information and the area of ​​the corresponding annotation are close to each other. If these areas are close to each other, the process proceeds to step S4217. If these areas are not close to each other, the process proceeds to step S4218.

[0261] In step S4217, the reliability determination unit 4102 determines that the reliability of the annotation recorded by the digital camera 100 is high, and refers to the focus level value recorded in the annotation as a key for the sorting process.

[0262] In step S4218 , the reliability determination unit 4102 is used to determine the level of detail of the comment assigned by the digital camera 100 .

[0263] In step S4219 , the reliability determination unit 4102 is used to determine the level of detail of the annotation assigned by the image processing application or the like.

[0264] In step S4220, the reliability determination unit 4102 determines whether the level of detail of the annotation assigned by the digital camera 100 is higher than the level of detail of the annotation assigned by the image processing application, etc. If the level of detail of the annotation assigned by the digital camera 100 is higher, the process proceeds to step S4217. If the level of detail of the annotation assigned by the digital camera 100 is lower, the process proceeds to step S4221.

[0265] In step S4221, the reliability determination unit 4102 determines whether the level of detail of the annotation assigned by the image processing application or the like is higher than the level of detail of the annotation assigned by the digital camera 100. If the level of detail of the annotation assigned by the image processing application or the like is higher, the process proceeds to step S4222. If the level of detail of the annotation assigned by the image processing application or the like is lower, the process proceeds to step S4223.

[0266] In step S4222, the reliability determination unit 4102 determines that the reliability of the annotation recorded by the image processing application or the like is high, and refers to the focus level value recorded in the annotation as a key for the sorting process.

[0267] In step S4223, it is determined that the reliability determination has failed using the reliability determination unit 4102. Then, a weighted average value of the respective rank values ​​recorded in the overlap annotation is calculated, and the calculated weighted average value is referenced as a key for the sorting process.

[0268] In step S4224, the control unit 201 performs the processing of steps S4201 to S4223 on all image files to be sorted to determine whether the rating value to be referenced has been determined. If the rating value to be referenced has been determined for all image files to be sorted, the process proceeds to step S4225. If there are image files for which the rating value to be referenced has not yet been determined, the process returns to step S4201, and the control unit 201 performs the processing of steps S4201 to S4223 on the image files to be sorted for which the rating value to be referenced has not yet been determined.

[0269] In step S4225, the control unit 201 sorts the image files to be sorted under the conditions determined by the user. The image files are sorted using information determined to be referenced by each image file through the processing of steps S4201 to S4223 among various information contained in the image files as a key.

[0270] In step S4226, the image display control unit 304 displays the sorting result on the display unit 206. The sorting result can be displayed so that multiple images are displayed in a list view, or a single image is displayed and an image feed operation is performed to sequentially display the sorted images (images found as search results). In addition, the comment display control unit 305 is used to display the comment determined by the reliability determination unit 4102 on the display unit 206.

[0271] According to the present embodiment, when images assigned with focus level values ​​as annotations in an overlapping manner are to be sorted on a thumbnail screen of an image display application or the like, values ​​to be referenced as keywords for the sorting process can be determined before the sorting process is performed.

[0272] Fifth embodiment

[0273] The fourth embodiment described above describes an example of sorting images assigned focus level values ​​as overlapping annotations on a thumbnail screen of an image display application. The fifth embodiment describes an example of filtering images assigned with subject-representing words as overlapping annotations on a thumbnail screen of an image display application. This embodiment describes an example of image filtering processing. However, any other processing, such as image search processing, may be performed. Since the basic configuration of this embodiment is the same as that of the fourth embodiment, its description will be omitted.

[0274] Figure 29 is a diagram showing an image to which overlapping words are assigned as annotations. Figure 29 An example is shown in which a subject name 5001 is assigned as a comment by the digital camera 100 at the time of image capturing, and a subject name 5002 is assigned as a comment by an image processing application or the like. Figure 29 , the captured image of “Tokyo Tower” is assigned a subject name 5001, which is a correct subject name (“Tokyo Tower”), and a subject name 5002, which is an incorrect subject name (“Eiffel Tower”).

[0275] When filtering images containing Figure 29 If the user filters the image of "Eiffel Tower" when multiple images are shown, then Figure 29 The image shown (ie, the image of Tokyo Tower) would incorrectly match the filtering result.

[0276] Will refer to Figure 26 、 Figure 30 and Figure 31 A method of using one correct subject name among overlapping subject names as a keyword for filtering processing is described.

[0277] Figure 26 1 is a diagram showing a processing unit included in a PC 200 according to a fifth embodiment of the present invention. In the fifth embodiment, a control unit 201 (processing unit) of the PC 200 includes Figure 26 The control unit 201 of the PC 200 functions as a computer including a computer system by loading a predetermined program stored in the ROM 202 into the work RAM and executing the program. Figure 26 The processing unit of the unit shown. The control unit 201 (processing unit) controls the program read by the control unit (processing unit) Figure 26 The units shown and Figure 2 The units of the PC 200 shown are implemented Figure 30 and Figure 31The flowchart shown in .

[0278] exist Figure 26 Among the units shown, units that perform the same processing as that in the fourth embodiment will not be described, and units that perform processing different from that of the fourth embodiment will be mainly described.

[0279] The annotation interpretation unit 4100 determines whether the acquired annotation can be interpreted as a subject name. In this embodiment, it is analyzed whether the annotation is in a suitable format that can be used as a keyword for image filtering processing.

[0280] The subject recognition information acquisition unit 4101 acquires subject recognition information recorded in an image recorded by the digital camera 100 during imaging, such as autofocus information, a depth map, and sensor information such as GPS information. In this embodiment, the subject recognition information acquisition unit 4101 acquires GPS information during imaging.

[0281] The reliability determination unit 4102 determines the reliability of the annotation using the reliability assigned to the annotation itself, the subject identification information acquired by the subject identification information acquisition unit 4101 , the level of detail of the annotation, the type of the annotation, and the like.

[0282] In this embodiment, when subject identification information is recorded in the target annotation, the reliability determination unit 4102 determines reliability using the subject identification information.

[0283] For example, the GPS information at the time of shooting is acquired by the subject recognition information acquisition unit 4101, and is combined with Figure 29 The comment corresponding to one of the shown subject names 5001 and 5002 and indicating a position geographically closer to the position indicated by the GPS information at the time of image capture is determined to have high reliability.

[0284] Furthermore, the level of detail of an annotation can be determined based on whether a more detailed subject name is assigned, using the subject name assigned to the target annotation. For example, when an annotation simply assigned the name "building" overlaps with an annotation assigned a building name such as "Tokyo Tower," the annotation assigned the building name (i.e., the more detailed subject name) can be determined to have a high reliability.

[0285] Furthermore, in this embodiment, if reliability cannot be determined even using the above information, a new annotation is generated from the overlapping annotations, and the reliability of the generated annotation is determined to be high. For example, consider a case where different subject names are assigned to the same subject (e.g., an image of "Tokyo Tower" is assigned the annotations "Tokyo Tower" and "Eiffel Tower"). In this case, the parent of the overlapping subject names ("Tokyo Tower" and "Eiffel Tower," that is, "Tower") is obtained and processed as an annotation with high reliability.

[0286] The annotation reference unit 4103 refers to the annotations used for image filtering among the overlapping annotations before performing image filtering based on the reliability determined by the reliability determination unit 4102. Highly reliable annotations and lowly reliable annotations can be referenced and displayed so that the user can understand the reliability level of the annotations.

[0287] Figure 30 and Figure 31 1 is a flowchart showing the operation procedure of the filtering process according to the fifth embodiment. In the image filtering process according to this embodiment, the subject name to be used as a keyword for the filtering process is determined for each image. Figure 27 and Figure 28 The steps described are similar to those described with Figure 27 and Figure 28 The same reference numerals as those shown in FIG are denoted, and their description will be omitted as appropriate. Figure 27 and Figure 28 Different processes in the process will be described with the same step number, but with filtering process instead. Figure 27 and Figure 28 The sorting process shown in .

[0288] In step S5104 , the annotation interpretation unit 4100 is used to analyze the subject name based on the overlapping annotation determined in step S4203 .

[0289] In step S5105, the control unit 201 determines whether to perform filtering processing in simple inspection mode or detailed inspection mode when filtering images. Simple inspection mode is a mode in which filtering processing is performed with reference to any one of the overlapping subject names. Detailed inspection mode is a mode in which filtering processing is performed using a plurality of subject names among the overlapping subject names. It is assumed that the user operating the image processing apparatus according to this embodiment has already selected simple inspection mode or detailed inspection mode in advance before giving an instruction to start filtering processing. It is also assumed that, when selecting simple inspection mode, the user has previously selected which subject name to prioritize among the overlapping subject names. If the user has selected simple inspection mode, the process proceeds to step S5106. If the user has selected detailed inspection mode, the process proceeds to step S5107.

[0290] In step S5106 , the control unit 201 refers to the subject name selected by the user in step S5105 among the overlapping subject names as a keyword for filtering processing.

[0291] In step S5107, the control unit 201 determines whether the overlapping subject names match. If the overlapping subject names match, the process advances to step S5108. If the overlapping subject names do not match, the process advances to step S5109.

[0292] In step S5108, the control unit 201 uses any one of the overlapping subject names that matches as a key for filtering processing.

[0293] In step S5109, the reliability determination unit 4102 refers to a subject name with higher reliability among the overlapping subject names as a keyword for filtering processing. Figure 31 Steps S4213 to S5123 in describe a method for determining reliability.

[0294] In step S5111 , the annotation interpretation unit 4100 is used to analyze the subject name based on the annotation determined in step S4203 .

[0295] In step S5112 , the control unit 201 refers to the subject name analyzed in step S5111 as a keyword for filtering processing.

[0296] In step S5114 , the reliability determination unit 4102 refers to the reliability recorded in the annotation itself among the overlapping annotations, and refers to the subject name written in the annotation with high reliability as a keyword for filtering processing.

[0297] In step S5115, it is determined whether the overlap annotation is assigned GPS information using the reliability determination unit 4102. If the overlap annotation is assigned GPS information, the process proceeds to step S5116. If not, the process proceeds to step S4218.

[0298] In step S5116, the reliability determination unit 4102 is used to determine that the subject name (e.g., annotation) among the overlapping annotations that represents a location geographically closer to the location indicated by the GPS information has higher reliability, and reference is made to the subject name as a keyword for filtering processing.

[0299] In step S5117, the reliability determination unit 4102 determines that the reliability of the annotation recorded by the digital camera 100 is high, and refers to the subject name recorded in the annotation as a keyword for filtering processing.

[0300] In step S5122 , the reliability determination unit 4102 determines that the reliability of the annotation recorded by the image processing application or the like is high, and refers to the subject name recorded in the annotation as a keyword for filtering processing.

[0301] In step S5123, it is determined that the reliability determination has failed using the reliability determination unit 4102. Then, a subject name indicating an upper level of the subject name recorded in the overlap annotation is obtained and referenced as a keyword for filtering processing.

[0302] In step S4224, the control unit 201 performs the processing of steps S4201 to S4221 and steps S5104 to S5123 on all image files to be filtered to determine whether a reference (keyword for filtering) has been determined. If a reference has been determined for all image files to be filtered, the process proceeds to step S4225. If there are image files for which a reference has not yet been determined, the process returns to step S4201, and the control unit 201 performs the processing of steps S4201 to S4221 and steps S5104 to S5123 on the image files to be filtered for which a reference (keyword for filtering) has not yet been determined.

[0303] In step S4225, the control unit 201 filters the image files to be filtered under the conditions determined by the user. The image files are filtered using information determined to be referenced among various information included in the image files as a keyword.

[0304] In step S4226, the image display control unit 304 displays the filtering result on the display unit 206. The filtering result may be displayed so that a plurality of images are displayed in a list view, or one image is displayed and an image feed operation is performed to sequentially display filtered images.

[0305] According to the present embodiment, when images assigned with subject names as overlapping annotations are to be filtered on a thumbnail screen of an image display application, etc., subject names to be referred to can be determined as keywords for the filtering process before the filtering process is performed.

[0306] Other embodiments

[0307] Although the present invention has been described in detail with reference to exemplary embodiments of the present invention, the present invention is not limited to these specific embodiments and can be modified in various ways without departing from the scope of the present invention. These modifications also fall within the scope of the present invention. The above embodiments are examples of the present invention and can be used in any combination.

[0308] The present invention can also be implemented by performing the following processing. That is, in this processing, the software (program) for implementing the functions of the above-mentioned embodiments is provided to a system or device via a network or various storage media, and the computer (or CPU, microprocessing unit (MPU), etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program are regarded as embodiments of the present invention.

[0309] The embodiments of the present invention may also be implemented by a computer of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more comprehensively referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the embodiments described above and / or includes one or more circuits (e.g., an application-specific integrated circuit (ASIC)) to perform the functions of one or more of the embodiments described above, and by a method executed by a computer of the system or device, for example, reading and executing computer-executable instructions from a storage medium to perform the functions of one or more of the embodiments described above and / or controlling one or more circuits to perform the functions of one or more of the embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)) and may include a network of separate computers or separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer from, for example, a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)). TM ), one or more of flash memory devices, memory cards, etc.

[0310] The embodiments of the present invention can also be implemented by the following method, that is, software (including computer program products of computer programs / instructions) that perform the functions of the above-mentioned embodiments is provided to a system or device through a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer program / instructions.

[0311] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: an acquisition unit configured to acquire an image file including a plurality of annotations; a determining unit configured to determine correlations between the plurality of annotations; as well as A control unit is configured to perform control to integrate the annotations determined to be relevant by the determination unit among the plurality of annotations into an integrated annotation and output the integrated annotation.

2. The image processing apparatus according to claim 1, wherein: Each of the plurality of annotations includes character string information and region information, and The determination unit determines a correlation based on the character string information and the area information.

3. The image processing apparatus according to claim 2, wherein: The determination unit determines that annotations having the same character string information among the plurality of annotations are related.

4. The image processing apparatus according to claim 2, wherein: The determination unit determines that annotations, among the plurality of annotations, which are related in terms of character string information and whose areas indicated by the area information overlap, are related.

5. The image processing apparatus according to claim 4, wherein: The determination unit determines that annotations, among the plurality of annotations, whose overlapping amount of the areas indicated by the area information is greater than or equal to a predetermined amount, are related.

6. The image processing apparatus according to claim 1, further comprising: a second determining unit configured to determine a metadata recording area in which the plurality of annotations are recorded in the image file; The control unit controls to integrate the annotations determined by the second determination unit to be recorded in different metadata recording areas among the plurality of annotations into an integrated annotation and output the integrated annotation.

7. The image processing apparatus according to claim 1, wherein: The control unit controls to output an annotation into which annotations determined to be related among the plurality of annotations included in the image file are integrated and annotations not integrated among the plurality of annotations.

8. The image processing apparatus according to claim 7, wherein: The control unit controls to display an annotation into which annotations determined to be related among the plurality of annotations included in the image file are integrated and annotations not integrated among the plurality of annotations together with image data of the image file.

9. The image processing apparatus according to claim 8, wherein: The control unit controls to display the annotations into which the annotations determined to be related are integrated and the annotations which are not integrated in different display forms.

10. The image processing apparatus according to claim 7, wherein: The control unit controls to generate an image file including an annotation into which annotations determined to be related among the plurality of annotations included in the image file are integrated and annotations not integrated among the plurality of annotations, and outputs the generated image file.

11. A method for controlling an image processing device, the method comprising: an acquisition step of acquiring an image file including a plurality of annotations; a determining step of determining the correlation between the plurality of annotations; as well as A control step of performing control to integrate the annotations determined to be relevant in the determination step among the plurality of annotations into an integrated annotation and output the integrated annotation. 12 . A computer-readable storage medium storing a program for causing a computer to function as the image processing apparatus according to claim 1 . 13 . A computer program product for causing a computer to function as the image processing apparatus according to claim 1 .

Images