Image processing apparatus and method for controlling the same, imaging apparatus, and program
Patent Information
- Application Number
- JP2021204438
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-12-16
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2041-12-16
AI Technical Summary
Existing technologies do not effectively manage information detected from input images, particularly the results of subject estimation in images.
An image processing apparatus that includes detection, classification, and management of subjects through inference processing, generating image files with inference results, and storing management information separately on a recording medium.
Enables effective management and utilization of estimated subject information in images, enhancing applications such as robot automation and autonomous driving.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus that processes an image captured by a digital camera or the like, a control method thereof, an imaging apparatus, and a program.
Background Art
[0002] In recent years, artificial intelligence (AI) technologies such as deep learning have been increasingly utilized in various technical fields. For example, conventionally, a digital still camera or the like has been equipped with a function of detecting a human face from a captured image. However, Patent Document 1 discloses a technique for accurately detecting animals such as dogs and cats as well as humans as detection targets and recognizing a subject.
[0003] On the other hand, as subject detection technology has evolved and the importance of the information that can be detected has increased, the input image and the detected result as detection targets have become important information that can be variously utilized in subsequent workflows. For example, various information estimated from an image using AI technology or the like has become very important input data in various fields such as robot automation and automobile automatic driving.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, Patent Document 1 does not touch on a method for managing information detected from an input image.
[0006] Therefore, the present invention provides a method for managing information detected from an input image, particularly an estimation result of a subject in the image.
Means for Solving the Problems
[0007] To achieve the above objective, the image processing apparatus of the present invention is A detection means for detecting a subject from an image, A processing means for classifying the detected subject by inference processing, A generation means for generating an image file containing the image data of the aforementioned image and the first inference result of the inference process, and recording it on a recording medium. The system includes storage means for storing management information, which includes information for identifying the image file and a second inference result of the inference process, in a management file different from the image file and recorded on the recording medium. [Effects of the Invention]
[0008] According to the present invention, a method for managing the estimation results of subjects in images can be provided. [Brief explanation of the drawing]
[0009] [Figure 1A] A block diagram showing the schematic configuration of the system in the embodiment. [Figure 1B] A block diagram showing the schematic configuration of the external display device 102 corresponding to the embodiment. [Figure 2] A flowchart showing an example of imaging processing corresponding to the embodiment. [Figure 3] A flowchart showing an example of the playback process corresponding to the embodiment. [Figure 4] A diagram showing an example of the data structure of image files and management files corresponding to the embodiment. [Figure 5] A flowchart showing an example of image transmission processing corresponding to the embodiment. [Modes for carrying out the invention]
[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.
[0011] Furthermore, in the following description, a digital camera is used as an example of an image processing device that uses an inference model to perform subject classification, but the embodiment of the image processing device is not limited to a digital camera. In this embodiment, the image processing device may be any device that reads an image recorded on a recording medium and displays it on a display device, such as a mobile phone, smartphone, tablet PC, or personal computer. Also, the image processing device is just one example of a name for a digital camera etc. corresponding to the embodiment, and may also be called an information processing device, image output device, image display device, image recording device, etc.
[0012] Figure 1A is a block diagram showing an example of a system configuration including a digital camera 100 according to an embodiment. In this embodiment, the system is configured such that the digital camera 100 is connected to an external device 101 and an external display device 102 in a manner that allows communication between them. The digital camera 100 is an image processing device as described above, and the external device 101 is a device that manages the inference model corresponding to this embodiment. The external display device 102 is an external device for displaying image files managed by the digital camera 100, and can be implemented as, for example, a personal computer, a tablet terminal, a smartphone, etc.
[0013] The configuration of the digital camera 100 is described below. First, the barrier 10 is a protective member that covers the imaging unit, including the imaging lens 11 of the digital camera 100, to prevent dirt and damage to the imaging unit, and its operation is controlled by the barrier control unit 43. The imaging lens 11 forms an optical image on the imaging surface of the image sensor 13. The shutter 12 has an aperture function. The image sensor 13 is composed of, for example, a CCD or a CMOS sensor, and converts the optical image formed on the imaging surface by the imaging lens 11 via the shutter 12 into an electrical signal.
[0014] The A / D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal converted by the A / D converter 15 is written to the memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on the information at the time of imaging and written to the memory 25. The development parameters consist of various parameters used in image processing to record JPEG format, such as exposure settings, white balance, color space, and contrast. The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50A and supplies clock signals and control signals to the image sensor 13, the A / D converter 15, and the D / A converter 21.
[0015] The image processing unit 20 performs various image processing operations, such as predetermined pixel interpolation, color conversion, correction, and resizing, on data from the A / D converter 15 or data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and calculation processing using the image data obtained from the image capture, and provides the obtained calculation results to the system control unit 50A. The system control unit 50A controls the exposure control unit 40 and the focus control unit 41 based on the provided calculation results to realize AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing.
[0016] Also, the image processing unit 20 performs predetermined arithmetic processing using the image data obtained by imaging, and also performs AWB (Auto White Balance) processing based on the obtained arithmetic result. Further, the image processing unit 20 reads the image data stored in the memory 25 and performs compression processing or decompression processing, such as JPEG format, MPEG-4 AVC or HEVC (High Efficiency Video Coding), and reversible compression for uncompressed RAW data. Then, the image processing unit 20 writes the processed image data into the memory 25.
[0017] Also, the image processing unit 20 performs predetermined arithmetic processing using the image data obtained by imaging, and performs editing processing of various image data. Specifically, it can perform trimming processing to adjust the display range and size of the image by making unnecessary parts around the image data invisible, and resizing processing to change the size by enlarging or reducing the image data or display elements of the screen. Further, it can perform RAW development to create image data by adding image processing such as color conversion to the data subjected to compression processing or decompression processing, such as reversible compression for uncompressed RAW data, and converting it to the JPEG format. Also, it can perform video extraction processing to extract a specified frame of a video format such as MPEG-4, convert it to the JPEG format, and save it.
[0018] Also, the image processing unit 20 also performs processing such as superimposing an OSD (On-Screen Display) such as a menu or arbitrary characters to be displayed on the display unit 23 together with the image data for display. Further, in the image processing unit 20, using the input image data, distance information from the subject obtained from the imaging element 13 at the time of imaging, etc., it performs subject detection processing to detect the subject existing in the image data and detect the subject area. As the detectable information, it is possible to obtain detection information such as the position and size in the image, regions such as these, inclination, and certainty.
[0019] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The RAW image data generated by the A / D converter 15 is written into the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22. The image data for display written into the image display memory 24 is displayed on the display unit 23 composed of a TFT LCD or the like via the D / A converter 21. By sequentially displaying the image data obtained by imaging using the display unit 23, it is possible to realize an electronic finder function for displaying a live image. The memory 25 has a storage capacity sufficient to store a predetermined number of still images or a moving image for a predetermined time, and stores the captured still images and moving images. Also, the memory 25 can be used as a work area for the system control unit 50A.
[0020] The exposure control unit 40 controls the shutter 12 having an aperture function. Also, the exposure control unit 40 has a flash light control function by interlocking with the flash 44. The focus control unit 41 performs focus adjustment by driving a focus lens (not shown) included in the imaging lens 11 based on an instruction from the system control unit 50A. The zoom control unit 42 controls zooming by driving a zoom lens (not shown) included in the imaging lens 11. The flash 44 has a function of projecting AF auxiliary light and a flash light control function.
[0021] The system control unit 50A controls the entire digital camera 100. The non-volatile memory 51 is an electrically erasable and recordable non-volatile memory, and for example, an EEPROM or the like is used. Note that the non-volatile memory 51 stores not only a program for executing processing corresponding to the present embodiment but also map information and the like.
[0022] The shutter switch 61 (SW1) turns ON during the operation of the shutter button 60 and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) turns ON when the operation of the shutter button 60 is completed and instructs the start of a series of imaging operations including exposure processing, development processing, and recording processing. In the exposure processing, the signal read from the image sensor 13 is written to the memory 25 as RAW image data via the A / D converter 15 and the memory control unit 22. In the development processing, the RAW image data written to the memory 25 is developed using calculations in the image processing unit 20 and the memory control unit 22, and written to the memory 25 as image data. In the recording processing, image data is read from the memory 25, compressed by the image processing unit 20, and after the compressed image data is stored in the memory 25, it is written to the external recording medium 91 via the card controller 90.
[0023] The control unit 63 is equipped with various buttons and touch panels. For example, it includes a power button, a menu button, a mode switch for switching between imaging mode, playback mode, and other special imaging modes, a directional pad, a set button, a macro button, and a multi-screen playback page change button. It also includes, for example, a flash setting button, a single-shot / continuous-shot / self-timer switching button, a menu navigation + (plus) button, a menu navigation - (minus) button, an image quality selection button, an exposure compensation button, and a date / time setting button.
[0024] The metadata generation and analysis unit 70 generates various metadata, such as the Exif (Exchangeable image file format) standard, to be attached to the image data when recording image data to the external recording medium 91, based on the information at the time of imaging. The Exif standard is a standard that defines the data format of image files, and more specifically, it is a standard that defines a data format for saving image data generated by the digital camera 100 together with various information (metadata) that represents the conditions and settings at the time of imaging. In addition, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata attached to the image data. Examples of metadata include imaging settings information at the time of imaging, image data information about the image data, and characteristic information of the subject included in the image data. Furthermore, when recording moving image data, the metadata generation and analysis unit 70 can also generate and attach metadata for each frame.
[0025] The power supply 80 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The power control unit 81 supplies power from the power supply 80 to each part of the digital camera 100. The card controller 90 transmits and receives data to and from an external recording medium 91 such as a memory card. The external recording medium 91 is, for example, a memory card and records images (still images and videos) captured by the digital camera 100.
[0026] The inference engine 73A performs inference on image data input via the system control unit 50A using the inference model recorded in the inference model recording unit 72A. The inference model can be one input from an external source such as an external device 101 via the communication unit 71A and recorded in the inference model recording unit 72A, or one obtained through retraining by the learning unit 74A. The inference model recording unit 72A maintains a management version so that each inference model can be identified when it is updated from an external source or through retraining by the learning unit 74A. Furthermore, the inference engine 73A includes a neural network design 73a.
[0027] The neural network design 73a has an intermediate layer (neurons) placed between the input layer and the output layer. Image data is input to the input layer from the system control unit 50A. Several layers of neurons are placed in the intermediate layer. The number of neuron layers and the number of neurons in each layer are determined as appropriate in the design. The intermediate layer is weighted based on the inference model recorded in the inference model recording unit 72A. Annotation information corresponding to the image data input to the input layer is output to the output layer.
[0028] In this embodiment, an inference model is assumed that infers the classification of subjects contained in an image. An inference model generated by deep learning using an external device 101 or the like is used, with image data of various subjects and their classification results (for example, classification of animals such as dogs and cats, or classification of subjects such as people, animals, plants, and buildings) as training data. However, since the inference engine 73A can be updated from the external device 101 or the like, it may also handle various other inference models.
[0029] The learning unit 74A retrains the inference model upon request from the system control unit 50A, etc. The learning unit 74A has a training data recording unit 74a, which records information about training data for the inference engine 73A. The learning unit 74A can retrain the inference engine 73A using the training data recorded in the training data recording unit 74a and update the inference engine 73A using the inference model recording unit 72A.
[0030] The communication unit 71A has a communication circuit for transmitting and receiving. Specifically, the communication circuit may be wireless communication such as Wi-Fi or Bluetooth (registered trademark), or wired communication such as Ethernet or USB. The communication unit 71A can communicate with the communication unit 71B of the external device 101 and the communication unit 71C of the external display device 102. The communication unit 71A functions as a communication unit that sends and receives various information to and from the system control unit 50A, system control unit 50B, and system control unit 50C, including not only image files with annotation information created by the inference engine 73A, but also inference models and training data. Furthermore, the information to be transmitted can be restricted depending on whether the external device 101 or the external display device 102 is associated with the digital camera 100.
[0031] The external device 101 includes a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system control unit 50B, and a communication unit 71B. Alternatively, it may be a device without these components. The learning unit 74B creates an inference model in response to requests from external sources such as the inference engine 73B and the system control unit 50B. The inference model recording unit 72B records the inference model transmitted from the digital camera 100 and the inference model created by the learning unit 74B.
[0032] Figure 1B shows an example of a more detailed configuration of the external display device 103. The external display device 103 includes an image processing unit 20C, a memory control unit 22C, a display unit 23C, an image display memory 24C, a memory 25C, a non-volatile memory 51C, an operation unit 62C, a communication unit 71C, a metadata analysis unit 75, a power supply 80C, and a power supply control unit 81C. The external display device 103 can receive image files etc. from the digital camera 100 via the communication unit 71C, and the system control unit 50C can display the received image files etc. on the display unit 23C.
[0033] Next, the imaging process of the digital camera 100 in this embodiment will be described using Figure 2. The process corresponding to the flowchart can be realized, for example, by the system control unit 50A executing the corresponding program (stored in the non-volatile memory 51, etc.) to control each functional block of the digital camera 100.
[0034] Processing begins when an imaging mode is selected using the mode selector switch included in the control unit 63. In S201, the system checks whether the user has pressed the shutter button 60, thereby turning on the shutter switches 61 (SW1) and 62 (SW2) and instructing still image capture. If still image capture is instructed, the system proceeds to S202.
[0035] In S202, imaging processing is performed. During imaging processing, the focus control unit 41 and exposure control unit 40 are used to perform AF (autofocus) processing and AE (automatic exposure) processing, and then the image signal output from the image sensor 13 via the A / D converter 15 is stored in the memory 25. Furthermore, the image processing unit 20 compresses the image signal stored in the memory 25 into JPEG or MPEG-4 HEVC format according to the user's settings.
[0036] In S203, the image processing unit 20 performs subject detection processing on the image signal stored in memory 25 and obtains detection information of subjects included in the image.
[0037] In S204, the image data created in S202 and the subject detection information acquired in S203 are recorded as an image file on the external recording medium 91. The image file recorded here is recorded in the format shown in Figure 4(A). The image file 400 recorded in this embodiment consists of an area for storing metadata in accordance with the Exif standard and an image data area for recording compressed image data 406. For example, if the user instructs the image file 400 to be recorded in JPEG format, it is recorded in JPEG format, and the Exif data 401 is recorded in the APP1 marker or the like. If the user instructs the image file 400 to be recorded in HEIF (High Efficiency Image File Format) format, it is recorded in HEIF file format, and the Exif data 401 is recorded in the Metadata Box or the like. Similarly, if the user instructs the image file to be recorded in RAW format, the Exif data 401 is recorded in a predetermined area such as the Metadata Box.
[0038] The subject detection information acquired by S203 is stored in MakerNote404, which is included in Exif data401, using the metadata generation and analysis unit 70. MakerNote404 can be recorded in a proprietary recording format of the digital camera 100 manufacturer (or vendor, hereafter the same), and its internal specifications and format are not publicly disclosed in the Exif standard, so only the manufacturer can understand its contents. Furthermore, MakerNote404 is an area that can only be used by vendor-specific software (for example, applications installed on personal computers, tablet devices, smartphones, or web services accessible from such devices). In addition, the subject detection information is recorded in annotation information 403, which is recorded at the position indicated by the annotation information offset included in the annotation link information storage tag 402. The subject detection information recorded in annotation information 403 is used as annotation information that becomes input data to the inference engine 73A in the inference process described later. In the example shown in Figure 5(A), only the coordinate area of the subject within the screen is recorded in annotation information 403, but other information may also be included.
[0039] In S205, it is checked whether inference processing using the inference engine 73A can be performed. For example, if the inference engine 73A is currently processing another image, or if the shutter switch 62 (SW2) remains ON due to the operation of the shutter button 60, and continuous shooting is instructed and imaging processing should take priority, then inference processing cannot be performed. If inference processing cannot be performed, the process proceeds to S208; if inference processing can be performed, the process proceeds to S206.
[0040] In S206, inference processing is performed on the image file 400 using the inference engine 73A. In this embodiment, the image file 400 is input to the inference engine 73A. The image data 406 and annotation information 403 within the image file 400 are used to identify the subject regions contained within the image data, and the classification results of the subjects contained within the subject regions are output as a result of inference performed using the inference engine 73A for each subject region. During inference, information related to the inference process, such as operational debug information and logs, may also be output in addition to the inference results.
[0041] In S207, the subject classification result, which is the inference result obtained in S206, is classified into a first inference result and a second inference result other than the first inference result according to predetermined criteria, the first inference result is recorded in an image file, and the process proceeds to S208. Figure 4(B) shows the image file 410 after the representative inference result for each subject has been recorded as the first inference result in the image file 400 of Figure 4(A). In the image file 410, the metadata generation and analysis unit 70 is used to record the representative inference result of a representative subject in the annotation information 411, and the representative inference result for each subject, including the main subject, is recorded in the annotation information 412 in MakerNote 404.
[0042] Here, the specified criteria can be, for example, whether the inference result is the most reliable, or whether the inference result relates to the main subject. The content of the specified criteria will be explained in detail below.
[0043] First, regarding reliability, the image processing unit 20 can determine the reliability of the inference result based at least partially on at least one of several imaging conditions, including the size of the subject area, blur of the subject area, whether the subject area is hidden or cut off by other subject areas, and exposure and brightness when the subject is imaged. For example, the reliability can be set with a maximum value of 100 and a minimum value of 0. Specifically, if the image is captured under imaging conditions suitable for classifying the subject, such as when the subject area is not blurred, not hidden or cut off by other subjects, and with appropriate exposure and brightness, the reliability of the inference result will be high. On the other hand, if the image is captured under imaging conditions unsuitable for classifying the subject, such as when the subject area is blurred, hidden or cut off by other subjects, under backlit conditions, or in a very dark environment, the reliability of the inference result will be low.
[0044] Next, regarding the main subject, the main subject and other subjects can be determined from among the subjects detected in the image data, based on at least one of the following: the size of the subject area in the image data, the size of a specific part of the subject, and the amount of displacement from the center position of the image. More specifically, the subject with the largest subject area in the image data, the subject with the largest specific part such as a face in the image data, or the subject with the smallest amount of displacement from the center position of the image data, i.e., the subject closest to the center position, can be designated as the main subject. Furthermore, subjects detected within an area specified by the user of the digital camera 100 may also be designated as the main subject.
[0045] In the example shown in Figure 4(B), multiple subjects, including at least subject A and subject B, are detected from image data 406. Subject A is determined to be the primary subject, and its representative inference result is recorded in annotation information 411. In addition, MakerNote 404 records the representative inference result for each of the multiple subjects detected from image data 406. In this way, the results of the inference process corresponding to know-how and assets can be protected by storing some of the information in an area that can be shared among manufacturers, where the description format is publicly available in the Exif standard, while storing the majority of the information in an area specific to the manufacturer, where the description format is not publicly available in the Exif standard, etc.
[0046] Next, in S208, the second inference result, which was not classified as the first inference result, is recorded in a management file on the external recording medium 91, which is a different file from the image file. If there is any management version or debug information of the current inference model held by the inference model recording unit 72A, these are recorded in the management file as inference model management information. While the image file is composed of separate files for each image, the management file can be configured to include information from multiple image files in a single file.
[0047] Since the management file is not the type of file that is created each time an image data file 406 is generated, it can be created in advance on an external storage medium 91, and corresponding information can be added each time an image file is created. In this case, it is also possible to create an independent management file for each user of the digital camera 100.
[0048] Furthermore, the management file can be recorded in the manufacturer's proprietary recording format, and since its internal specifications and format are not publicly disclosed under DCF standards, Exif standards, etc., it can be created in a way that only the manufacturer can understand its contents. In addition, the management file can be made accessible (viewable) only through the manufacturer's official software (for example, an application installed on a personal computer, tablet device, or smartphone, or a web service accessible from such devices).
[0049] In the subsequent S209, if the user signals that imaging is complete, such as by turning off the power using the mode switch or power button included in the control unit 63, the imaging process is terminated.
[0050] Figure 4 shows an example of the data structure of the image file and management file corresponding to this embodiment. Figure 4(A) shows an example of the image file structure when, in S204, the image data created in S202 and the subject detection information acquired in S203 are recorded as an image file on the external recording medium 91. Figure 4(B) shows an example of the image file structure of image file 410 after a portion of the inference results have been recorded in the image file 400 of Figure 4(A). Figure 4(C) shows an example of the structure of the management file 420 which records the inference results and inference model management information, etc.
[0051] In Figure 4(B), the image file 410 uses the metadata generation and analysis unit 70 to record representative inference results for the main subject in annotation information 411, associated with the subject's position coordinates. Since this annotation information 411 is recorded in the format specified by the Exif standard, it can be used regardless of the manufacturer of the digital camera 100. In addition, representative inference results for the main subject and other subjects are recorded as annotation information 412 in MakerNote 404 for each subject's position coordinates. In this embodiment, the inference results recorded in the image file 410 are collectively referred to as the first inference results.
[0052] The management file 420 shown in Figure 4(C) stores management information 421A to 421N individually for n (n is a natural number of 1 or more than 2) image files. Here, management file 421A refers to the management file corresponding to image file 410 shown in Figure 4(B). Image files corresponding to other management files 421B, etc., exist in a similar format, but are omitted in Figure 4. Management information 421A to 421N records image information 422A to 422N for identifying the corresponding image file 410. Image information 422 can include, for example, the image file number, the date and time of acquisition, the image format, etc., and the corresponding image file 410 can be identified by the image file number. Also, in Figure 4(C), an example is shown where all inference results, including the first inference result recorded in image file 410, are recorded in annotation information 423A to 423N for each subject, associated with its position coordinates. Furthermore, inference model management information 424 is also recorded in each management information 421.
[0053] In this embodiment, annotation information 411, 412, and 423 are shown as examples of a combination of position coordinates and inference results, but the annotation information only needs to record the output of the inference result of the inference model. For example, the output result can be recorded as is, and the recording format and type are not limited to text format, binary format, etc.
[0054] Thus, the image file 410 records image data 406 and some of the inference results from among multiple inference results (for example, an inference result that represents multiple inference results). Therefore, thereafter, the image data 406 and the inference results can be efficiently associated and easily managed without having to manage them separately. Furthermore, by recording only representative inference results, the process of generating the image file 400 can be simplified, and the file size can be reduced. In addition, since the inference model management information 424 is unique data used by the manufacturer that manages the inference model, it is not made widely available to the public. By recording it in a non-public management file 420, it can be safely managed in association with the inference results.
[0055] In the above explanation, representative inference results are recorded in an image file, and all inference results are recorded in a management file. However, if the predetermined criterion is whether or not the result is an inference result for the main subject, then all inference results for the main subject may be recorded in the image file, and all inference results for all subjects may be recorded in the management file. In this case, it is advisable to record a representative inference result for the main subject in annotation information 411 and all inference results for the main subject in annotation information 412 within the image file. The representative inference result in this case should be determined by whether it is the inference result with the highest confidence level.
[0056] Furthermore, the annotation information 412 in the image file and the annotation information 413 in the management file may not record all inference results, but only the differences with annotation information 411 and annotation information 412. Also, instead of judging based on confidence level or whether it is the main subject, predetermined items from the inference results may be used as representative inference results. For example, the representative inference result may be information indicating whether the subject is a person or an animal, while other inference results may be predetermined as information such as the subject's name, gender, and age.
[0057] Next, with reference to Figure 3, the playback process of the digital camera 100 corresponding to the embodiment will be described. The process corresponding to the flowchart can be realized, for example, by the system control unit 50A executing the corresponding program (stored in non-volatile memory 51, etc.) and controlling each functional block of the digital camera 100.
[0058] In the digital camera 100, when the playback mode is selected using the mode switch included in the control unit 63, playback processing to view image data recorded on the external recording medium 91 is initiated. Once playback processing begins, in S301, one image file to be played back is identified and read from the external recording medium 91 to the memory 25 via the card controller 90.
[0059] In S302, the image processing unit 20 performs decompression and resizing processing on the image file read into the memory 25 according to the recording method, converts it into display image data, records it in the image display memory 24, and plays it back on the display unit 23.
[0060] In S303, the metadata generation and analysis unit 70 checks whether the inference results for the currently playing image file have been recorded in the management file 420. If they have been recorded, the process proceeds to S304; otherwise, it proceeds to S312. In S304, if the user instructs the system to display the inference results recorded in the image file using the operating components included in the operation unit 63, the process proceeds to S305; otherwise, the process proceeds to S310.
[0061] In S305, the metadata generation and analysis unit 70 analyzes the management file 420, identifies the management information of the currently playing image file from the management information 421, obtains annotation information 423, and proceeds to S306. In S306, the metadata generation and analysis unit 70 extracts information related to the inference results from the annotation information 423, and uses the image processing unit 20 to superimpose the extracted information onto the image data being played and display it. For example, the inference results that classify the subjects within the coordinate regions are displayed as text together with a frame indicating the coordinate region for each subject described in the annotation information 423, so that the user can visually see the inference results associated with the image data. After that, the process proceeds to S310.
[0062] In S307, the same processing as in S206 is performed, and the process proceeds to S308. In S308, similar to S207, the subject classification results, which are the inference results obtained in S307, are classified into a first inference result and a second inference result that does not fall under the first inference result. The first inference result is recorded in image file 410, and then the process proceeds to S309. In S309, similar to S208, the second inference result that was not classified under the first inference result, image information, inference model management information, etc., are recorded in a management file 420, which is a different file from image file 410. At this time, the first inference result is also recorded in the management file. After that, the process proceeds to S310.
[0063] In S310, if an instruction to play another image is given using the operating member included in the operating unit 63, the process returns to S301 and the above-described process is repeated for the instructed playback image. On the other hand, if no instruction to play another image is given, in S311 it is determined whether an instruction to end the playback process has been given. If an instruction to end the playback process has not been given, the process returns to S303 and the above-described process is repeated. If an instruction to end the playback process has been given, the playback process is terminated.
[0064] In this embodiment, by recording only the first inference result, which is a representative inference result, in the image file 410, the processing of the metadata generation and analysis unit 70 when displaying the inference result is reduced, and processing speed can be increased. If more detailed inference results are required, the management file 420 can be used. Since the management file 420 contains management information 421 for multiple image files, the metadata generation and analysis unit 70 can obtain and use the inference results for multiple image files at once by analyzing a single management file 420.
[0065] Next, with reference to Figure 5, the transmission process of the digital camera 100 in this embodiment will be described. Here, we will explain using the example where the image file before transmission is recorded in the format of the image file 410 described above, and the management file 420 is also recorded.
[0066] When transmission is instructed using the operating element included in the operation unit 63, the transmission process shown in the flowchart of Figure 5 is initiated. The transmission instruction may be initiated by selecting one or more image files as targets, or it may be initiated when an image file is generated, after setting up automatic transmission in advance during image capture. Here, we will explain using the case where the image files before transmission are recorded in the format of the image file 410 in Figure 4(B) and the management file 420 in Figure 4(C) as described above. The process corresponding to the flowchart in Figure 5 can be realized, for example, by the system control unit 50A executing the corresponding program (stored in the non-volatile memory 51, etc.) and controlling each functional block of the digital camera 100.
[0067] In Figure 5, in S501, the digital camera 100 requests system information from the external display device 102 via the communication unit 71A, and the external display device 102 transmits the system information from the system control unit 50C to the digital camera 100 via the communication unit 71C. As a result, the digital camera 100 can obtain the system information from the external display device 102. The process then proceeds to S502.
[0068] In S502, the system control unit 50A determines the method of transmission to the external display device 102. Specific examples of transmission methods include wireless transmission such as Wi-Fi or Bluetooth®, wired transmission via Ethernet cable or USB cable, and transmission via removable media such as an SD card. As for how to determine the transmission method, if multiple transmission methods are available, the user may set the transmission method using the operating components included in the operation unit 63, or it may be determined from the connection status between the external device 101 and the digital camera 100. If only one transmission method is available, that transmission method may be determined.
[0069] In the subsequent S503, the system control unit 50A determines from the system information of the external display device 102 acquired in S501 whether the external display device 102 can access the management file 420. As mentioned above, the management file 420 is recorded in a manufacturer's proprietary description format that is not publicly available in the Exif standard, etc., and can only be accessed by specific applications or web services. Here, we assume that the external display device 102 is a tablet terminal, and that an application provided by the manufacturer of the digital camera 100, or an application compatible thereto, is installed on the tablet terminal. In this case, the digital camera 100 and the external display device 102 are connected using this application, and the system information provided by the external display device 102 includes information indicating that the application is installed. Since the management file 420 can be accessed using this application, if the application is installed, it can be determined that the management file 420 is available (accessible).
[0070] In addition to the above, the determination of whether the management file 420 is available may be based on whether a response is returned when a predetermined communication command (for example, a manufacturer-specific communication command) is sent from the digital camera 100 to the external display device 102. Furthermore, the determination may be based on received system information, such as whether a user has been registered or whether they have been charged. Regarding user registration, the management file 420 may be determined to be available only if it matches the user information of the digital camera 100. This helps protect the management file 420 in cases where a third party tries to access it using an application provided by the manufacturer in order to misuse the digital camera 100.
[0071] If it is determined that the management file 420 is available, the process proceeds to S504 and executes the transmission process for the management file 420. On the other hand, if it is not available, the transmission process is skipped and the process proceeds to S505.
[0072] In the subsequent S504, the system control unit 50A transmits the target management file via the communication unit 71A using the transmission method determined in S502 for the external display device 102, and the process then proceeds to S505. The external display device 102 analyzes the management file received from the communication unit 71C using the metadata analysis unit 75 in the same manner as in S502, and the inference results can be displayed on the display unit 23C. In S505, the system control unit transmits the target image file using the transmission method determined in S502, and the process ends.
[0073] In this transmission process, by sending management files only to specific recipients, it is possible to prevent the leakage of know-how and assets (data created at cost) to unspecified recipients. <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices (e.g., a host computer, interface devices, scanners, video cameras, etc.) or to a device consisting of a single device (e.g., a copier, a facsimile machine, etc.).
[0074] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0075] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]
[0076] Digital camera: 100, External device: 101, External display device: 102
Claims
1. A detection means for detecting a subject from an image, A processing means for classifying the detected subject by inference processing, A generation means for generating an image file containing the image data of the aforementioned image and the first inference result of the inference process, and recording it on a recording medium. A storage means for storing management information, including information for identifying the image file and the second inference result of the inference process, in a management file different from the image file, which is recorded on the recording medium. An image processing apparatus having
2. The image processing apparatus according to claim 1, wherein the first inference result is a predetermined inference result among the inference results obtained by the inference process, and the second inference result is an inference result other than the first inference result among the inference results obtained by the inference process.
3. The image processing apparatus according to claim 2, wherein the first inference result is an inference result selected from the inference results of the inference process according to a predetermined criterion, and the second inference result is an inference result other than the first inference result among the inference results of the inference process.
4. The first inference result is recorded in the image file in a predetermined format. The image processing apparatus according to any one of claims 1 to 3, wherein the management file is a file in a proprietary recording format of the manufacturer of the image processing apparatus.
5. The image processing apparatus according to any one of claims 1 to 4, wherein the management information further includes information of an inference model used by the processing means for the inference processing.
6. The image processing apparatus according to claim 3, wherein the predetermined criterion is that the reliability of the inference result is the highest.
7. The confidence level is determined at least in part based on the imaging conditions of the subject in the image subjected to the inference process. The image processing apparatus according to claim 6, wherein the reliability of a subject is determined to be higher the subject that is imaged under imaging conditions suitable for the classification of the subject.
8. The image processing apparatus according to claim 7, wherein the imaging conditions include at least one of the following: the size of the subject area of the subject, blur of the subject, whether the subject is obscured by another subject, whether the subject is not cut off, exposure at the time of imaging, and brightness.
9. The image processing apparatus according to any one of claims 3, 6 to 8, wherein the predetermined criterion is the inference result of the inference process for the main subject among the detected subjects.
10. The image processing apparatus according to claim 9, wherein the main subject is determined from among the detected subjects based on at least one of the following: the size of the subject area in the image, the size of a specific part of the subject, and the amount of displacement of the subject from the center position of the image.
11. The image processing apparatus according to claim 9, wherein the main subject is a subject detected within a specified range in the image.
12. The aforementioned image file is The first inference result of the main subject is stored in a data area that conforms to a predetermined standard format. The image processing apparatus according to any one of claims 9 to 11, wherein the first inference results of other subjects among the detected subjects, other than the main subject, are stored in a data area in a format unique to the manufacturer of the image processing apparatus.
13. The image processing apparatus according to any one of claims 1 to 12, wherein the management information further includes the first inference result.
14. The image processing apparatus according to any one of claims 1 to 13, wherein the management file includes the management information associated with each of the multiple image files.
15. A determination means for determining whether an external display device connected to the image processing device can use the management file, A transmission means for transmitting the image file to the external display device. It further possesses, If the determination means determines that the external display device can use the management file, the transmission means transmits the image file and the management file to the external display device. If the determination means determines that the external display device cannot use the management file, the transmission means transmits only the image file to the external display device. The image processing apparatus according to any one of claims 1 to 14.
16. The image processing apparatus according to claim 15, wherein the determination means determines that the management file is available when the external display device uses predetermined software capable of referencing the management file.
17. The image processing apparatus according to claim 15, wherein the determination means determines that the management file is available when the external display device returns a response to a predetermined command.
18. The image processing apparatus according to claim 15, wherein the determination means determines that the management file is available when a predetermined user registration has been performed on the external display device.
19. An imaging means for capturing the aforementioned image, The image processing apparatus according to any one of claims 1 to 18 and An imaging device characterized by having the following features.
20. The detection means includes a detection step of detecting a subject from an image, The processing means includes a processing step of classifying the detected subject by inference processing, The generation means includes a generation step of generating an image file containing the image data of the image and the first inference result of the inference process and recording it on a recording medium, Storage means includes a storage step of storing management information, which includes information for identifying the image file and the second inference result of the inference process, in a management file different from the image file and recorded on the recording medium, A method for controlling an image processing device, including the control method for an image processing device.
21. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 18.
