Image processing device and control method thereof, imaging device, and program

Abstract

To manage a result of estimating a subject in an image.SOLUTION: An image processing apparatus has: detection means that detects a subject from an image; processing means that classifies the detected subject through inference processing; creation means that creates an image file including image data of the image and a first inference result of the inference processing and records the file in a recording medium; and storage means that stores management information including information for specifying the image file and a second inference result of the inference processing in a management file that is different from the image file and that is recorded in the recording medium.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present invention relates to an image processing device for processing images captured by a digital camera or the like, a control method for the same, an imaging device, and a program. [Background technology]

[0002] In recent years, artificial intelligence (AI) technologies such as deep learning have been utilized in various technical fields. For example, conventionally, digital still cameras and the like have been equipped with a function for detecting human faces from captured images, but Patent Document 1 discloses a technology for accurately detecting and recognizing subjects, not limited to humans, but also animals such as dogs and cats.

[0003] On the other hand, as subject detection technology evolves and the information that can be detected becomes more important, the images input as detection targets and the detection results become important information that can be utilized in various ways in subsequent workflows. For example, the diverse information that can be inferred from images using AI technology is extremely important input data in various fields, such as robot automation and self-driving cars. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-99559 Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 does not mention a method for managing information detected from an input image.

[0006] Therefore, the present invention provides a method for managing information detected from an input image, in particular, estimation results of a subject in the image. [Means for solving the problem]

[0007] In order to achieve the above object, the image processing device of the present invention comprises: a detection means for detecting a subject from an image; a processing means for classifying the detected subject by an inference process; a generating means for generating an image file including image data of the image and a first inference result of the inference processing and recording the image file on a recording medium; a storage means for storing management information including information for identifying the image file and a second inference result of the inference process in a management file that is different from the image file and is recorded on the recording medium; death, the first inference result is an inference result selected from the inference results of the inference processing in accordance with a predetermined criterion, and the second inference result is an inference result other than the first inference result from among the inference results of the inference processing, the predetermined criterion is that the reliability of the inference result is the highest; The reliability is determined based at least in part on an image capturing condition of a subject in the image that is the subject of the inference process; The subject imaged under imaging conditions suitable for the classification of the subject is determined to have a higher reliability. . [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a method for managing estimation results of a subject in an image. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1 is a block diagram showing a schematic configuration of a system according to an embodiment. [Figure 1B] FIG. 2 is a block diagram showing a schematic configuration of an external display device 102 according to the embodiment. [Figure 2] 10 is a flowchart showing an example of an imaging process according to the embodiment. [Figure 3] 10 is a flowchart showing an example of a reproduction process according to the embodiment. [Figure 4] 3A and 3B are diagrams showing examples of data configurations of an image file and a management file according to the embodiment. [Figure 5] 10 is a flowchart showing an example of an image transmission process according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following description, a digital camera is used as an example of an image processing device that uses an inference model to classify objects, but embodiments of the image processing device are not limited to digital cameras. In this embodiment, the image processing device may be any device that reads images recorded on a recording medium and displays them on a display device, such as a mobile phone, smartphone, tablet PC, or personal computer. The image processing device is an example of a name for a digital camera or the like corresponding to the embodiment, and may also be called an information processing device, image output device, image display device, image recording device, or the like.

[0012] 1A is a block diagram showing an example of a system configuration including a digital camera 100 according to an embodiment. In this embodiment, the system is configured such that the digital camera 100 is communicably connected to an external device 101 and an external display device 102. As described above, the digital camera 100 is an image processing device, and the external device 101 is a device that manages an inference model corresponding to this embodiment. The external display device 102 is an external device for displaying image files managed by the digital camera 100, and can be realized as, for example, a personal computer, a tablet terminal, a smartphone, or the like.

[0013] The configuration of the digital camera 100 will be described below. First, the barrier 10 is a protective member that covers the imaging section including the imaging lens 11 of the digital camera 100 to prevent the imaging section from getting dirty or damaged, and its operation is controlled by a barrier control section 43. The imaging lens 11 forms an optical image on the imaging surface of the imaging element 13. The shutter 12 has an aperture function. The imaging element 13 is composed of, for example, a CCD or CMOS sensor, and converts the optical image formed on the imaging surface by the imaging lens 11 via the shutter 12 into an electrical signal.

[0014] The A / D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal converted by the A / D converter 15 is written to memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on information at the time of image capture and written to memory 25. The development parameters consist of various parameters used in image processing for recording in JPEG format, such as exposure setting, white balance, color space, and contrast. The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50A, and supplies clock signals and control signals to the image sensor 13, the A / D converter 15, and the D / A converter 21.

[0015] The image processing unit 20 performs various image processing such as predetermined pixel interpolation, color conversion, correction, and resizing on data from the A / D converter 15 or data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and arithmetic processing using image data obtained by capturing an image, and provides the obtained arithmetic results to the system control unit 50A. The system control unit 50A controls the exposure control unit 40 and focus control unit 41 based on the provided arithmetic results, thereby realizing AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing.

[0016] The image processing unit 20 also performs predetermined calculations using the captured image data and performs AWB (auto white balance) processing based on the calculation results. Furthermore, the image processing unit 20 reads image data stored in the memory 25 and performs compression or decompression processing, such as JPEG, MPEG-4 AVC, or HEVC (High Efficiency Video Coding), or lossless compression of uncompressed RAW data. The image processing unit 20 then writes the processed image data to the memory 25.

[0017] The image processing unit 20 also performs predetermined arithmetic processing using the captured image data, editing various types of image data. Specifically, it can perform a trimming process to adjust the display range and size of the image by hiding unnecessary parts around the image data, and a resizing process to change the size of the image data or screen display elements by enlarging or reducing them. Furthermore, it can perform RAW development, which applies image processing such as color conversion to data that has been compressed or expanded, such as lossless compression of uncompressed RAW data, and converts it to JPEG format to create image data. It can also perform video clipping, which clips out specified frames from video formats such as MPEG-4, converts them to JPEG format, and saves them.

[0018] The image processing unit 20 also performs processing such as superimposing an OSD (On-Screen Display) such as a menu or arbitrary characters to be displayed on the display unit 23 together with the image data for display. Furthermore, the image processing unit 20 performs subject detection processing to detect a subject present in the image data and detect the subject area by using the input image data and information such as the distance to the subject obtained from the image sensor 13 at the time of image capture. Detectable information includes area such as position and size within the image, as well as detection information such as inclination and likelihood.

[0019] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The RAW image data generated by the A / D converter 15 is written to the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22. The display image data written to the image display memory 24 is displayed on the display unit 23, which may be configured as a TFT LCD or the like, via the D / A converter 21. By sequentially displaying captured image data on the display unit 23, an electronic viewfinder function that displays live images can be realized. The memory 25 has a storage capacity sufficient to store a predetermined number of still images or a predetermined period of moving images, and stores captured still images and moving images. The memory 25 can also be used as a work area for the system control unit 50A.

[0020] The exposure control unit 40 controls the shutter 12, which has an aperture function. The exposure control unit 40 also has a flash dimming function by working in conjunction with the flash 44. The focus control unit 41 adjusts focus by driving a focus lens (not shown) included in the imaging lens 11 based on instructions from the system control unit 50A. The zoom control unit 42 controls zooming by driving a zoom lens (not shown) included in the imaging lens 11. The flash 44 has an AF assist light projection function and a flash dimming function.

[0021] The system control unit 50A controls the entire digital camera 100. The nonvolatile memory 51 is an electrically erasable and recordable nonvolatile memory, such as an EEPROM. The nonvolatile memory 51 stores not only programs for executing the processes according to this embodiment, but also map information and the like.

[0022] The shutter switch 61 (SW1) is turned on during operation of the shutter button 60, and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) is turned on when the shutter button 60 is completely operated, and instructs the start of a series of imaging operations including exposure processing, development processing, and recording processing. In the exposure processing, a signal read from the image sensor 13 is written to the memory 25 as RAW image data via the A / D converter 15 and the memory control unit 22. In the development processing, the RAW image data written to the memory 25 is developed using calculations in the image processing unit 20 and the memory control unit 22, and written to the memory 25 as image data. In the recording processing, the image data is read from the memory 25, compressed by the image processing unit 20, and stored in the memory 25 before being written to the external recording medium 91 via the card controller 90.

[0023] The operation unit 63 includes various operation members such as buttons and a touch panel. For example, it includes a power button, a menu button, a mode switch for switching between image capture mode, playback mode, and other special image capture modes, a cross key, a set button, a macro button, and a multi-screen playback page break button. It also includes, for example, a flash setting button, a single / continuous / self-timer switching button, a menu navigation + (plus) button, a menu navigation - (minus) button, an image capture quality selection button, an exposure compensation button, and a date / time setting button.

[0024] When recording image data on the external recording medium 91, the metadata generation and analysis unit 70 generates various metadata, such as Exif (Exchangeable Image File Format) standard metadata, to be attached to the image data based on information at the time of capture. The Exif standard defines the data format of image files, and more specifically, defines a data format for attaching various information (metadata) indicating the circumstances and settings at the time of capture to image data generated by the digital camera 100 and saving the image data together as a file. Furthermore, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata attached to the image data. Examples of metadata include capture setting information at the time of capture, image data information related to the image data, and feature information of the subject included in the image data. Furthermore, when recording moving image data, the metadata generation and analysis unit 70 can generate and attach metadata for each frame.

[0025] The power supply 80 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. A power supply control unit 81 supplies power from the power supply 80 to each unit of the digital camera 100. A card controller 90 transmits and receives data to and from an external recording medium 91 such as a memory card. The external recording medium 91 may be, for example, a memory card, and records images (still images and videos) captured by the digital camera 100.

[0026] The inference engine 73A uses an inference model recorded in the inference model recording unit 72A to perform inference on image data input via the system control unit 50A. The inference model may be input from an external device such as the external device 101 via the communication unit 71A and recorded in the inference model recording unit 72A, or may be an inference model obtained by re-learning using the learning unit 74A. Note that the inference model maintains a management version in the inference model recording unit 72A or the like so that each inference model can be identified when it is updated from the outside or by re-learning using the learning unit 74A. The inference engine 73A also has a neural network design 73a.

[0027] In the neural network design 73a, an intermediate layer (neurons) is arranged between the input layer and the output layer. Image data is input to the input layer from the system control unit 50A. Several layers of neurons are arranged in the intermediate layer. The number of neuron layers is determined appropriately in the design, and the number of neurons in each layer is also determined appropriately in the design. The intermediate layer is weighted based on the inference model recorded in the inference model recording unit 72A. Annotation information corresponding to the image data input to the input layer is output to the output layer.

[0028] In this embodiment, an inference model is assumed that infers the classification of a subject included in an image. Image data of various subjects and the classification results (for example, classification of animals such as dogs and cats, or classification of subjects such as people, animals, plants, and buildings) are used as training data, and an inference model generated by deep learning using an external device 101 or the like is used. However, since the inference engine 73A can be updated from the external device 101 or the like, various other inference models may also be used.

[0029] The learning unit 74A re-learns the inference model upon receiving a request from the system control unit 50A, etc. The learning unit 74A has a teacher data recording unit 74a, which records information about the teacher data for the inference engine 73A. The learning unit 74A re-learns the inference engine 73A using the teacher data recorded in the teacher data recording unit 74a, and can update the inference engine 73A using the inference model recording unit 72A.

[0030] The communication unit 71A has a communication circuit for transmitting and receiving data. Specifically, the communication circuit may be a wireless communication circuit such as Wi-Fi or Bluetooth (registered trademark), or a wired communication circuit such as Ethernet or USB. The communication unit 71A can communicate with the communication unit 71B of the external device 101 and the communication unit 71C of the external display device 102. The communication unit 71A functions as a communication unit that transmits and receives not only image files with annotation information created by the inference engine 73A but also various information such as inference models and training data to and from the system control unit 50A, the system control unit 50B, and the system control unit 50C. In addition, the information to be transmitted can be restricted depending on whether the external device 101 or the external display device 102 is associated with the digital camera 100.

[0031] The external device 101 has a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system control unit 50B, and a communication unit 71B. Alternatively, the external device 101 may not have these components. The learning unit 74B creates an inference model upon receiving a request from an external device, such as the inference engine 73B or the system control unit 50B. The inference model recording unit 72B records the inference model sent from the digital camera 100 and the inference model created by the learning unit 74B.

[0032] 1B shows an example of a more detailed configuration of the external display device 103. The external display device 103 has an image processing unit 20C, a memory control unit 22C, a display unit 23C, an image display memory 24C, a memory 25C, a non-volatile memory 51C, an operation unit 62C, a communication unit 71C, a metadata analysis unit 75, a power supply 80C, and a power supply control unit 81C. The external display device 103 can receive image files and the like from the digital camera 100 via the communication unit 71C, and display the image files and the like received by the system control unit 50C on the display unit 23C.

[0033] Next, the imaging process of the digital camera 100 in this embodiment will be described with reference to Fig. 2. The process corresponding to this flowchart can be realized, for example, by the system control unit 50A executing a corresponding program (stored in the non-volatile memory 51 or the like) and controlling each functional block of the digital camera 100.

[0034] The process starts when an imaging mode is instructed using a mode changeover switch included in the operation unit 63. In S201, it is confirmed whether the user has pressed the shutter button 60, turning on the shutter switches 61 (SW1) and 62 (SW2) to instruct still image capture, and if still image capture has been instructed, the process proceeds to S202.

[0035] In S202, imaging processing is performed. In the imaging processing, AF (autofocus) processing and AE (autoexposure) processing are performed using the focus control unit 41 and exposure control unit 40, and the image signal output from the image sensor 13 via the A / D converter 15 is then stored in memory 25. Furthermore, the image signal stored in memory 25 is compressed in JPEG format or MPEG-4 HEVC format using compression processing included in the image processing unit 20 in accordance with user settings to create image data.

[0036] In S203, the image processing unit 20 performs subject detection processing on the image signal stored in the memory 25, and obtains detection information of the subject included in the image.

[0037] In S204, the image data created in S202 and the subject detection information acquired in S203 are recorded as an image file on the external recording medium 91. The image file recorded here is recorded in the format shown in FIG. 4A. The image file 400 recorded in this embodiment is composed of an area for storing at least metadata conforming to the Exif standard and an image data area for recording compressed image data 406. For example, if the user instructs recording in JPEG format, the image file 400 is recorded in JPEG format, and Exif data 401 is recorded in an APP1 marker or the like. If the user instructs recording in HEIF (High Efficiency Image File Format) format, the image file 400 is recorded in HEIF file format, and the Exif data 401 is recorded in a Metadata Box or the like. Similarly, if recording in RAW format is instructed, the Exif data 401 is recorded in a predetermined area such as a Metadata Box.

[0038] The subject detection information acquired in S203 is stored in the MakerNote 404 included in the Exif data 401 by the metadata generation and analysis unit 70. The MakerNote 404 can be recorded in a recording format unique to the manufacturer (or vendor, hereinafter the same) of the digital camera 100, and since its internal specifications and format are not disclosed in the Exif standard, only the manufacturer itself can understand its contents. Furthermore, the MakerNote 404 is an area that can only be used by genuine software from the vendor (for example, applications installed on personal computers, tablet devices, or smartphones, or web services accessible on such devices). Furthermore, the subject detection information is recorded in the annotation information 403 recorded at the position indicated by the annotation information offset included in the annotation link information storage tag 402. The subject detection information recorded in the annotation information 403 is used as annotation information that serves as input data to the inference engine 73A in the inference processing described below. 4In the example shown in (A), only the coordinate area of ​​the subject within the screen is recorded in the annotation information 403, but other information may also be included.

[0039] In S205, it is confirmed whether inference processing using the inference engine 73A can be performed. For example, if the inference engine 73A is currently processing another image, or if the shutter switch 62 (SW2) is still ON due to operation of the shutter button 60, and continuous shooting is instructed and image capture processing should take priority, the inference processing cannot be performed. If inference processing cannot be performed, S 209 If the inference process is executable, the process proceeds to S206.

[0040] In S206, inference processing is performed on the image file 400 using the inference engine 73A. In this embodiment, the image file 400 is input as input to the inference engine 73A. The subject areas contained in the image data are identified from the image data 406 and annotation information 403 in the image file 400, and a classification result of the subjects contained in the subject areas is output as a result of inference performed for each subject area using the inference engine 73A. During the inference, in addition to the inference results, information related to the inference processing, such as debug information on the operation during the inference, logs, etc., may be output.

[0041] In S207, the subject classification results, which are the inference results obtained in S206, are classified into first inference results and second inference results other than the first inference results according to predetermined criteria, the first inference results are recorded in the image file, and the process proceeds to S208. FIG. 4B shows an image file 410 after recording representative inference results for each subject as the first inference results in the image file 400 of FIG. 4A. In the image file 410, the metadata generation and analysis unit 70 records representative inference results for representative subjects in annotation information 411, and records representative inference results for each subject, including the main subject, in annotation information 412 in the MakerNote 404.

[0042] Here, the predetermined criterion may be, for example, whether the inference result has the highest reliability or whether the inference result is for the main subject. The specific contents of the predetermined criterion will be described below.

[0043] First, regarding the reliability, the image processing unit 20 can determine the reliability of the inference result based at least in part on at least one of a plurality of imaging conditions, including the size of the subject area, blur of the subject area, whether the subject area is obscured or cut off by other subject areas, and the exposure and brightness at the time of capturing the subject. The reliability can be, for example, 100 at its highest and 0 at its lowest. Specifically, if the subject area is captured under imaging conditions suitable for classifying the subject, such as when the subject area is not blurred or obscured or cut off by other subjects and is captured with appropriate exposure and brightness, the reliability of the inference result will be high. On the other hand, if the subject area is captured under imaging conditions unsuitable for classifying the subject, such as when the subject area is blurred, obscured or cut off by other subjects, backlit, or captured in a very dark environment, the reliability of the inference result will be low.

[0044] Next, for the main subject, the main subject and subjects other than the main subject can be determined from the subjects detected from the image data based on at least one of the size of the subject area in the image data, the size of a specific part of the subject, and the amount of displacement from the center position of the image. More specifically, the subject with the largest subject area in the image data, the subject with the largest specific part such as a face in the image data, or the subject with the smallest amount of displacement from the center position of the image data, i.e., the subject closest to the center position, can be determined as the main subject. Furthermore, the subject detected within an area specified by the user of digital camera 100 may be determined as the main subject.

[0045] 4(B), multiple subjects including at least subjects A and B are detected from image data 406, of which subject A is determined to be the main subject, and a representative inference result is recorded in annotation information 411. Also, in MakerNote 404, a representative inference result for each of the multiple subjects detected from image data 406 is recorded. In this way, with regard to the results of inference processing that correspond to know-how or assets, some of the information can be stored in an area that can be shared between manufacturers whose description format has been made public in the Exif standard or the like, while the majority of the information can be stored and protected in an area specific to the manufacturer whose description format has not been made public in the Exif standard or the like.

[0046] Next, in S208, the second inference result that was not classified as the first inference result is recorded in a management file, which is a file different from the image file, on the external recording medium 91. If there is a management version or debug information of the current inference model held in the inference model recording unit 72A, this is recorded in the management file as inference model management information. While image files are configured as independent files for each image, management files can be configured to include information on multiple image files in a single file.

[0047] The management file is not the type of file that is created each time image data 406 is generated, so it can be created in advance on the external recording medium 91, and corresponding information can be added each time an image file is created. In this case, an independent management file can also be created for each user who uses the digital camera 100.

[0048] Furthermore, management files can be recorded in a recording format unique to the manufacturer, and can be created so that only the manufacturer itself can understand their contents, since their internal specifications and format are not made public by the DCF standard, Exif standard, etc. Furthermore, management files can be made available (accessible) only with specified software genuine to the manufacturer (for example, an application installed on a personal computer, tablet device, or smartphone, or a web service accessible on such device, etc.).

[0049] In the next step S209, when the user issues an instruction to complete the imaging process by turning off the power using the mode changeover switch or power button included in the operation unit 63, the imaging process ends.

[0050] Fig. 4 shows an example of the data structure of an image file and a management file according to this embodiment. Fig. 4(A) shows an example of the structure of an image file when, in S204, the image data created in S202 and the subject detection information acquired in S203 are recorded as an image file on the external recording medium 91. Fig. 4(B) shows an example of the structure of an image file 410 after recording a portion of the inference results in the image file 400 of Fig. 4(A). Fig. 4(C) shows an example of the structure of a management file 420 in which the inference results, inference model management information, etc. are recorded.

[0051] In Fig. 4(B), the image file 410 uses the metadata generation and analysis unit 70 to record a representative inference result for the main subject in annotation information 411 in association with the position coordinates of the subject. Because the annotation information 411 is recorded in a format defined by the Exif standard, it can be used regardless of the manufacturer of the digital camera 100. Furthermore, representative inference results for the main subject and other subjects are recorded as annotation information 412 in MakerNote 404 for each position coordinate of the subject. In this embodiment, the inference results recorded in the image file 410 are collectively referred to as first inference results.

[0052] The management file 420 shown in Fig. 4(C) stores management information 421A to 421N for n (n is a natural number of 1 or 2 or more) image files. Here, the management file 421A indicates the management file corresponding to the image file 410 shown in Fig. 4(B). information Image files corresponding to 421B, etc. also exist in a similar format, but are omitted from Fig. 4. Management information 421A to 421N record image information 422A to 422N for identifying the corresponding image file 410. The image information 422 may include, for example, the image file number, the date and time of capture, and the image format, and the corresponding image file 410 can be identified based on the image file number. Fig. 4(C) also shows an example in which all inference results, including the first inference result recorded in the image file 410, are recorded in annotation information 423A to 423N for each subject, in association with their position coordinates. Inference model management information 424 is also recorded in each management information 421.

[0053] In this embodiment, the annotation information 411, 412, and 423 are an example of a combination of position coordinates and inference results, but the annotation information may be any information that is output as an inference result of an inference model. For example, the output result may be recorded as is, or the recording format or format may be any format, such as text or binary.

[0054] In this way, the image file 410 records the image data 406 and some of the inference results (for example, an inference result that represents the plurality of inference results), so that the image data 406 and the inference results can be efficiently associated and easily managed thereafter without having to be managed individually. Furthermore, by recording only the representative inference results, the image file 410This simplifies the generation process and reduces the file size. Furthermore, since the inference model management information 424 is specific data used by the manufacturer that manages the inference model, it is not made publicly available, but is recorded in a private management file 420, allowing it to be safely managed in association with the inference results.

[0055] In the above explanation, a representative inference result is recorded in an image file, and all inference results are recorded in a management file. However, if the predetermined criterion is whether or not the inference result is for the main subject, all inference results for the main subject may be recorded in an image file, and all inference results for all subjects may be recorded in a management file. In this case, it is preferable that the image file records a representative inference result for the main subject in annotation information 411, and records all inference results for the main subject in annotation information 412. In this case, it is preferable to determine the representative inference result based on whether it is the inference result with the highest reliability.

[0056] Also, annotation information 412 in the image file and annotation information in the management file 423 In the annotation information 410, not all inference results may be recorded, but only the differences from the annotation information 411 and the annotation information 412 may be recorded. Also, predetermined items among the inference results may be used as representative inference results, without determining the reliability or whether the subject is the main subject, etc. For example, the representative inference result may be information indicating whether the subject is a person or an animal, and other inference results may be predetermined information such as the subject's name, gender, age, etc.

[0057] Next, the playback process of the digital camera 100 according to the embodiment will be described with reference to Fig. 3. The process according to the flowchart can be realized, for example, by the system control unit 50A executing a corresponding program (stored in the non-volatile memory 51, etc.) and controlling each functional block of the digital camera 100.

[0058] In the digital camera 100, when the playback mode is instructed by the mode selector switch included in the operation unit 63, a playback process is started to view image data recorded on the external recording medium 91. When the playback process is started, in S301, one image file to be played back is identified and read from the external recording medium 91 into the memory 25 via the card controller 90.

[0059] In S302, the image processing unit 20 performs expansion processing, resizing processing, etc. on the image file read into the memory 25 according to the recording method, converts it into image data for display, records it in the image display memory 24, and plays it back on the display unit 23.

[0060] In S303, the metadata generation and analysis unit 70 checks whether the inference result of the currently played image file has been recorded in the management file 420. If it has been recorded, the process proceeds to S304, and if it has not been recorded, the process proceeds to S305. S307 In S304, if the user issues an instruction using an operation member included in the operation unit 63 to display the inference results recorded in the image file, the process proceeds to S305, and if no instruction is issued, the process proceeds to S310.

[0061] In S305, the metadata generation and analysis unit 70 analyzes the management file 420, identifies the management information of the currently played image file from the management information 421, obtains the annotation information 423, and proceeds to S306. In S306, the metadata generation and analysis unit 70 extracts information related to the inference result from the annotation information 423, and uses the image processing unit 20 to superimpose and display the extracted information on the image data being played. For example, the inference result associated with the image data can be displayed so that the user can visually confirm it by displaying a frame indicating the coordinate area of ​​each subject described in the annotation information 423 together with a character string indicating the inference result classifying the subjects within that coordinate area. Then, the process proceeds to S310.

[0062] In S307, the same processing as in S206 is performed, and the process proceeds to S308. In S308, as in S207, the subject classification results, which are the inference results obtained in S307, are classified into a first inference result and a second inference result other than the first inference result, and the first inference result is recorded in an image file 410, after which the process proceeds to S309. In S309, as in S208, the second inference result not classified into the first inference result, image information, inference model management information, etc. are recorded in a management file 420, which is a file different from the image file 410. At this time, the first inference result is also recorded in the management file. Thereafter, the process proceeds to S310.

[0063] In S310, if an instruction to play another image is given using an operating member included in the operation unit 63, the process returns to S301 and the above-described process is repeated for the instructed image to be played. On the other hand, if an instruction to play another image is not given, the process determines in S311 whether an instruction to end the playback process has been given, and if an instruction to end the playback process has not been given, the process returns to S303 and the above-described process is repeated, and if an instruction to end the playback process has been given, the playback process ends.

[0064] In this embodiment, by recording only the first inference result, which is a representative inference result, in the image file 410, the processing by the metadata generation and analysis unit 70 when displaying the inference result is reduced, enabling faster processing. If more detailed inference results are required, the management file 420 can be used. Since management information 421 for multiple image files is recorded in the management file 420, the metadata generation and analysis unit 70 can obtain and use inference results for multiple image files at once by analyzing one management file 420.

[0065] Next, the transmission process of the digital camera 100 in this embodiment will be described with reference to Fig. 5. Here, the image file before transmission is recorded in the format of the image file 410 described above, and the case where the management file 420 is also recorded will be described as an example.

[0066] When a transmission instruction is issued using an operating member included in the operation unit 63, the transmission process shown in the flowchart of Fig. 5 is initiated. The transmission instruction may be issued by selecting one or more image files as targets and initiating transmission, or may be set in advance to automatically transmit images when they are captured, triggering transmission when the image files are generated. Here, an example will be described in which the image files before transmission are recorded in a format such as the image file 410 in Fig. 4(B) and the management file 420 in Fig. 4(C) described above. The processing corresponding to the flowchart of Fig. 5 can be implemented, for example, by the system control unit 50A executing a corresponding program (stored in the non-volatile memory 51, etc.) and controlling each functional block of the digital camera 100.

[0067] 5, in S501, the digital camera 100 requests system information from the external display device 102 via the communication unit 71A, and the external display device 102 transmits the system information from the system control unit 50C to the digital camera 100 via the communication unit 71C. This allows the digital camera 100 to obtain the system information of the external display device 102. Thereafter, the process proceeds to S502.

[0068] In S502, the system control unit 50A determines a transmission method for the external display device 102. Specific examples of transmission methods include wireless transmission such as Wi-Fi or Bluetooth (registered trademark), wired transmission using an Ethernet cable or USB cable, and transmission using removable media such as an SD card. When multiple transmission methods are available, the transmission method may be determined by a method set by the user using an operation member included in the operation unit 63, or may be determined based on the connection status between the external device 101 and the digital camera 100. When only a single transmission method is available, that transmission method may be determined.

[0069] In the next step S503, the system control unit 50A determines whether the external display device 102 can use the management file 420 based on the system information of the external display device 102 acquired in step S501. As described above, the management file 420 is recorded in a manufacturer-specific description format that is not publicly disclosed in the Exif standard or the like, and therefore can only be referenced by a specific application or web service. Here, it is assumed that the external display device 102 is a tablet terminal, and that an application provided by the manufacturer of the digital camera 100 or an application compatible with the application is installed on the tablet terminal. In this case, the digital camera 100 and the external display device 102 are connected using the application, and the system information provided by the external display device 102 includes information indicating that the application has been installed. Because the management file 420 can be referenced using the application, it can be determined that the management file 420 is available (can be referenced) if the application has been installed.

[0070] In addition to the above, whether the management file 420 is available for use may be determined based on whether a response is returned when a predetermined communication command (for example, a communication command specific to the manufacturer) is sent from the digital camera 100 to the external display device 102. Furthermore, the determination may be based on information such as whether the user has been registered or whether payment has been made based on received system information. Regarding user registration, the management file 420 may be compared with the user information of the digital camera 100, and it may be determined that the management file 420 is available for use only if it matches the user information. This is useful for protecting the management file 420 in the event that a third party accesses the digital camera 100 using an application provided by the manufacturer or the like in order to fraudulently use the digital camera 100.

[0071] If it is determined that the management file 420 is available, the process proceeds to S504 and executes the transmission process of the management file 420. On the other hand, if it is not available, the transmission process is skipped and the process proceeds to S505.

[0072] In the next step S504, the system control unit 50A performs a process of transmitting the management file to the external display device 102 via the communication unit 71A using the transmission method determined in step S502, and then the process proceeds to step S505. The external display device 102 analyzes the management file received from the communication unit 71C using the metadata analysis unit 75. S305 By analyzing the data in the same manner as above, the inference result can be displayed on the display unit 23C. In S505, the transmission process of the target image file is performed using the transmission method determined in S502, and the process ends.

[0073] In this way, in the above-described transmission process, by transmitting the management file only to specific destinations, it is possible to prevent the leakage of know-how and assets (data created at great expense) to unspecified destinations. <Other embodiments> The present invention may be applied to a system consisting of multiple devices (e.g., a host computer, an interface device, a scanner, a video camera, etc.), or to an apparatus consisting of a single device (e.g., a copier, a facsimile machine, etc.).

[0074] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] Digital camera: 100, external device: 101, external display device: 102

Claims

1. a detection means for detecting a subject from an image; a processing means for classifying the detected subject by an inference process; a generating means for generating an image file including image data of the image and a first inference result of the inference processing and recording the image file on a recording medium; a storage means for storing management information including information for identifying the image file and a second inference result of the inference process in a management file that is different from the image file and is recorded on the recording medium, the first inference result is an inference result selected from the inference results of the inference processing in accordance with a predetermined criterion, and the second inference result is an inference result other than the first inference result from among the inference results of the inference processing, the predetermined criterion is that the reliability of the inference result is the highest; The reliability is determined based at least in part on an image capturing condition of a subject in the image that is the subject of the inference process; The image processing device determines that the subject captured under imaging conditions that are more suitable for the classification of the subject has a higher reliability.

2. 2. The image processing device according to claim 1, wherein the imaging conditions include at least one of a size of a subject area of ​​the subject, blur of the subject, whether the subject is hidden by another subject, whether the subject is cut off, exposure at the time of imaging, and brightness.

3. A detection means for detecting a subject from an image; a processing means for classifying the detected subject by an inference process; a generating means for generating an image file including image data of the image and a first inference result of the inference processing and recording the image file on a recording medium; a storage means for storing management information including information for identifying the image file and a second inference result of the inference process in a management file that is different from the image file and is recorded on the recording medium, the first inference result is an inference result selected from the inference results of the inference processing in accordance with a predetermined criterion, and the second inference result is an inference result other than the first inference result from among the inference results of the inference processing, the predetermined criterion is an inference result of the inference process for a main subject among the detected subjects; Image processing device.

4. 4. The image processing device according to claim 3, wherein the main subject is determined from among the detected subjects based on at least one of a size of a subject area in the image, a size of a specific part of the subject, and an amount of displacement of the subject from a center position of the image.

5. The image processing device according to claim 3 , wherein the main subject is a subject detected within a specified range in the image.

6. The image file is storing the first inference result of the main subject in a data area conforming to a predetermined standard format; The image processing device according to claim 3 , wherein the first inference results of the detected subjects other than the main subject are stored in a data area conforming to a format unique to a manufacturer of the image processing device.

7. the first inference result is recorded in the image file in a format of a predetermined standard; The image processing device according to claim 1 , wherein the management file is a file in a recording format unique to a manufacturer of the image processing device.

8. The image processing device according to claim 1 , wherein the management information further includes information on an inference model used by the processing means for the inference processing.

9. The image processing device according to claim 1 , wherein the management information further includes the first inference result.

10. The image processing device according to claim 1 , wherein the management file includes the management information associated with each of a plurality of image files.

11. A detection means for detecting a subject from an image; a processing means for classifying the detected subject by an inference process; a generating means for generating an image file including image data of the image and a first inference result of the inference processing and recording the image file on a recording medium; a storage means for storing management information including information for identifying the image file and a second inference result of the inference process in a management file that is different from the image file and is recorded on the recording medium; a determination means for determining whether an external display device connected to the image processing device can use the management file; a transmitting means for transmitting the image file to the external display device; and when the determining means determines that the external display device can use the management file, the transmitting means transmits the image file and the management file to the external display device; When the determining means determines that the management file cannot be used by the external display device, the transmitting means transmits only the image file to the external display device.

12. The image processing device according to claim 11 , wherein the determining unit determines that the management file is available when the external display device uses predetermined software that can refer to the management file.

13. 12. The image processing device according to claim 11, wherein said determining means determines that said management file is available when said external display device returns a response to a predetermined command.

14. 12. The image processing device according to claim 11, wherein the determining means determines that the management file is available when a predetermined user registration has been made on the external display device.

15. imaging means for capturing the image; The image processing device according to any one of claims 1 to 14, An imaging device having the above configuration.

16. a detection step in which a detection means detects a subject from an image; a processing step in which a processing means classifies the detected subject by an inference process; a generation step in which a generation means generates an image file including image data of the image and a first inference result of the inference processing, and records the image file on a recording medium; a storing step in which a storing unit stores management information including information for identifying the image file and a second inference result of the inference processing in a management file that is different from the image file and is recorded on the recording medium, the first inference result is an inference result selected from the inference results of the inference processing in accordance with a predetermined criterion, and the second inference result is an inference result other than the first inference result from among the inference results of the inference processing, the predetermined criterion is that the reliability of the inference result is the highest; The reliability is determined based at least in part on an image capturing condition of a subject in the image that is the subject of the inference process; A control method for an image processing device, wherein a subject captured under imaging conditions that are more suitable for the classification of the subject is determined to have a higher reliability.

17. A detection step in which a detection means detects a subject from an image; a processing step in which a processing means classifies the detected subject by an inference process; a generation step in which a generation means generates an image file including image data of the image and a first inference result of the inference processing, and records the image file on a recording medium; a storing step in which a storing unit stores management information including information for identifying the image file and a second inference result of the inference process in a management file that is different from the image file and is recorded on the recording medium; a determining step in which a determining means determines whether an external display device connected to the image processing device can use the management file; a transmitting step in which a transmitting means transmits the image file to the external display device; and If it is determined in the determining step that the external display device can use the management file, the image file and the management file are transmitted to the external display device in the transmitting step; If it is determined in the determining step that the external display device cannot use the management file, only the image file is transmitted to the external display device in the transmitting step.

18. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 14.

Images