Image processing apparatus, control method thereof, and storage medium

Abstract

The present disclosure relates to an image processing apparatus, a control method thereof, and a storage medium. The image processing apparatus includes a detection unit configured to detect a subject from an image, a processing unit configured to classify the detected subject by performing an inference process, a generation unit configured to generate an image file including image data of the image and a first inference result of the inference process, and record the image file in a recording medium, and a storage unit configured to store management information including information for specifying the image file and a second inference result of the inference process in a management file different from the image file and recorded in the recording medium.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for processing images captured using a digital camera or the like, a control method for the image processing apparatus, and a non-transitory computer-readable storage medium. Background Technology

[0002] In recent years, artificial intelligence (AI) technologies, such as deep learning, have begun to be used in various technical fields. For example, functions for detecting faces from captured images are commonly installed in digital cameras. However, Japanese Patent Application Publication No. 2015-99559 discloses a technology in which the detection object is not limited to humans, but accurately detects animals such as dogs and cats to identify the subject.

[0003] On the other hand, with advancements in subject detection technology and the subsequent increase in the importance of detectable information, images as input to the objects being detected and the results derived from their detection have become important as information that can be used in various ways in subsequent workflows. For example, various types of information estimated from images using AI technologies now play a very important role as input data in various fields, such as robotics automation and autonomous vehicle driving.

[0004] However, Japanese Patent Application Publication No. 2015-99559 does not disclose any method for managing information detected from an input image.

[0005] In view of this, the present invention provides a method for managing information detected from an input image, and in particular a method for managing estimation results of a subject in an image. Summary of the Invention

[0006] One aspect of the embodiments relates to an image processing apparatus, comprising: a detection unit configured to detect a subject from an image; a processing unit configured to classify the detected subject by performing inference processing; a generation unit configured to generate an image file including image data of the image and a first inference result of the inference processing, and to record the image file in a recording medium; and a storage unit configured to store management information including information for specifying the image file and a second inference result of the inference processing in a management file different from the image file and recorded in the recording medium.

[0007] Another aspect of the embodiment relates to a control method for an image processing apparatus, the control method comprising: a detection step for detecting a subject from an image; a processing step for classifying the detected subject by performing inference processing; a generation step for generating an image file including image data of the image and a first inference result of the inference processing, and recording the image file in a recording medium; and a storage step for storing management information including information for specifying the image file and a second inference result of the inference processing in a management file different from the image file and recorded in the recording medium.

[0008] Another aspect of the embodiments relates to a non-transitory computer-readable storage medium storing one or more programs, the programs comprising instructions that, when executed by a processor of an image processing device, cause the image processing device to function as: a detection unit configured to detect a subject from an image; a processing unit configured to classify the detected subject by performing inference processing; a generation unit configured to generate an image file including image data of the image and a first inference result of the inference processing, and to record the image file in a recording medium; and a storage unit configured to store management information including information for specifying the image file and a second inference result of the inference processing in a management file different from the image file and recorded in the recording medium.

[0009] Other features of the invention will become apparent from the following description of exemplary embodiments (with reference to the accompanying drawings). Attached Figure Description

[0010] Figure 1A This is a block diagram illustrating a schematic configuration of the system in the embodiment.

[0011] Figure 1B This is a block diagram illustrating a schematic configuration of an external display device 102 according to an embodiment.

[0012] Figure 2 This is a flowchart illustrating an example of camera processing according to an embodiment.

[0013] Figure 3 This is a flowchart illustrating an example of playback processing according to an embodiment.

[0014] Figure 4A This is a diagram illustrating an example of the data configuration of an image file according to an embodiment.

[0015] Figure 4B This is a diagram illustrating another example of the data configuration of an image file according to an embodiment.

[0016] Figure 4C This is a diagram illustrating an example of the data configuration of a management file according to an embodiment.

[0017] Figure 5 This is a flowchart illustrating an example of image transmission processing according to an embodiment. Detailed Implementation

[0018] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but this does not limit the invention to requiring all such features, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0019] Furthermore, although the digital camera, as an imaging device, will be described in the following description as an example implementation of an image processing device that uses an inference model to classify subjects, the image processing device is not limited to being embodied in a digital camera. In this embodiment, the image processing device can be any image processing device that reads an image recorded on a recording medium and displays the image on a display device, and can be, for example, a mobile phone, smartphone, tablet PC, or personal computer. Furthermore, "image processing device" is an example of a name that can be used to refer to a digital camera or the like according to the embodiment, and the digital camera or the like according to the embodiment can also be referred to as an "information processing device," "image output device," "image display device," "image recording device," etc.

[0020] Figure 1A This is a block diagram illustrating a configuration example of a system including a digital camera 100 according to an embodiment. In this embodiment, the system is formed by communicatively connecting the digital camera 100 to various of an external device 101 and an external display device 102. The digital camera 100 is an image processing device as described above, and the external device 101 is a device for managing an inference model according to this embodiment. The external display device 102 is an external device for displaying image files managed by the digital camera 100, and can be implemented using, for example, a personal computer, a tablet terminal, or a smartphone.

[0021] The configuration of the digital camera 100 will now be described. First, the baffle 10 is a protective member that covers the imaging unit of the digital camera 100, including the imaging lens 11, and thereby prevents the imaging unit from being contaminated or damaged. The operation of the baffle 10 is controlled by the baffle control unit 43. The imaging lens 11 forms an optical image on the imaging surface of the image sensor 13. The shutter 12 has an aperture function. The image sensor 13 is, for example, composed of a CCD, CMOS sensor, etc., and converts the optical image formed on the imaging surface by the imaging lens 11 through the shutter 12 into an electrical signal.

[0022] The A / D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal obtained by the A / D converter 15 is written into the memory 25 as so-called raw image data. Additionally, display parameters corresponding to each piece of raw image data are generated based on information acquired during image capture, and these display parameters are written into the memory 25. The display parameters consist of various types of parameters used in image processing for recording in formats such as JPEG, including exposure settings, white balance, color space, and contrast. The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50A, and provides clock and control signals to the image sensor 13, the A / D converter 15, and the D / A converter 21.

[0023] The image processing unit 20 performs various types of image processing on data from the A / D converter 15 or the memory control unit 22, such as predetermined pixel interpolation, color conversion, correction, and resizing. Furthermore, the image processing unit 20 performs predetermined image processing and calculations using the captured image data and provides the obtained calculation results to the system control unit 50A. The system control unit 50A controls the exposure control unit 40 and the focus control unit 41 based on the provided calculation results to implement autofocus (AF), auto exposure (AE), and pre-flash (EF) processing.

[0024] Furthermore, the image processing unit 20 performs predetermined calculations using the captured image data and performs automatic white balance (AWB) processing based on the obtained calculation results. Additionally, the image processing unit 20 reads the image data stored in the memory 25 and performs compression or decompression processing, such as compression or decompression of JPEG, MPEG-4 AVC, or High Efficiency Video Coding (HEVC) formats, or lossless compression of the uncompressed raw data. Then, the image processing unit 20 writes the processed image data into the memory 25.

[0025] Furthermore, the image processing unit 20 performs predetermined computational processing using the captured image data and edits various types of image data. Specifically, the image processing unit 20 can perform cropping processing, which adjusts the displayed image area and size by setting unnecessary parts around the image data as non-display parts, and resizing processing, which changes the size by enlarging and shrinking the image data, screen display elements, etc. Additionally, the image processing unit 20 can perform raw image processing, in which data that has undergone compression or decompression processing (such as lossless compression of uncompressed raw data) undergoes further image processing (such as color conversion), and then converts it to JPEG format to create image data. Furthermore, the image processing unit 20 can perform motion picture extraction processing, in which specified frames of motion picture formats such as MPEG-4 are extracted, converted to JPEG format, and stored.

[0026] Furthermore, the image processing unit 20 performs on-screen display (OSD) processing, which overlays arbitrarily defined characters and menus to be displayed on the display unit 23 onto the displayed image data. Additionally, the image processing unit 20 performs subject detection processing, where subject distance information, such as that obtained from the image sensor 13 or the like during image capture, is used to detect subjects present in the image data and to detect the subject region of the subject. The detection information that can be obtained includes the region of the subject (including the position and size of the subject in the image), the tilt of the subject, the probability of the subject, etc.

[0027] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. Raw image data generated by the A / D converter 15 is written to the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22. The display image data written to the image display memory 24 is displayed on the display unit 23, which is composed of a TFT LCD or the like, via the D / A converter 21. An electronic viewfinder function for displaying real-time images can be implemented by sequentially displaying the captured image data using the display unit 23. The memory 25 is provided with sufficient storage capacity for storing a predetermined number of still images or moving images of a predetermined duration, and stores both captured still images and moving images. Furthermore, the memory 25 can also be used as the working area of ​​the system control unit 50A.

[0028] Exposure control unit 40 controls shutter 12, which has an aperture function. Furthermore, exposure control unit 40 also has a flash adjustment function implemented in cooperation with flash unit 44. Focus control unit 41 adjusts the focus by driving a focusing lens (not shown) included in imaging lens 11, based on instructions from system control unit 50A. Zoom control unit 42 controls zoom by driving a zoom lens (not shown) included in imaging lens 11. Flash unit 44 has AF assist light projection function and flash adjustment function.

[0029] The system control unit 50A controls the entire digital camera 100. The non-volatile memory 51 is an electrically erasable and recordable non-volatile memory, and for example, an EEPROM is used. Note that the non-volatile memory 51 contains not only one or more programs for performing the processing according to this embodiment, but also map information, etc.

[0030] When the shutter button 60 is partially actuated, the shutter switch 61 (SW1) is activated, providing instructions to begin operations such as AF processing, AE processing, AWB processing, and EF processing. When the shutter button 60 is fully actuated, the shutter switch 62 (SW2) is activated, providing instructions to begin a series of imaging operations including exposure processing, image processing, and recording processing. In exposure processing, the signal read from the image sensor 13 is written to the memory 25 as raw image data via the A / D converter 15 and the memory control unit 22. In image processing, the raw image data written to the memory 25 is developed using calculations performed by the image processing unit 20 and the memory control unit 22, and then written to the memory 25 as image data. In recording processing, image data is read from the memory 25 and compressed by the image processing unit 20, and the compressed image data is stored in the memory 25 and then written to the external recording medium 91 via the card controller 90.

[0031] The operation unit 63 includes operation components such as various types of buttons and touch panels. For example, the operation unit 63 includes a power button, a menu button, a mode switch for switching between video recording mode, playback mode, and other special video recording modes, arrow keys, a settings button, a macro button, and a multi-screen playback pagination button. For example, the operation unit 63 also includes a flash setting button, a single shot / continuous shot / Selfie switch button, a menu shift + (add) button, a menu shift - (subtract) button, a video image quality selection button, an exposure correction button, a date / time setting button, etc.

[0032] When image data is recorded on the external recording medium 91, the metadata generation and analysis unit 70 generates various types of metadata, conforming to the Exif (Exchangeable Image File Format) standard, to be added to the image data based on information acquired during recording. The Exif standard defines the data format of image files, and specifically, it defines a data format for adding various types of information (metadata) indicating recording conditions, settings, etc., to the image data generated by the digital camera 100 and storing the image data and information (metadata) together as a file. Furthermore, when reading image data recorded on the external recording medium 91, the metadata generation and analysis unit 70 analyzes the metadata attached to the image data. Examples of metadata include, for example, recording setting information applied during recording, image data information related to the image data, and feature information of the subject included in the image data. Additionally, when recording moving image data, the metadata generation and analysis unit 70 can also generate and attach metadata for each frame.

[0033] The power supply 80 is composed of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The power control unit 81 supplies power from the power supply 80 to the components of the digital camera 100. The card controller 90 sends data to and receives data from an external recording medium 91, such as a memory card. The external recording medium 91 is, for example, a memory card, and records images (still images, moving images) captured by the digital camera 100.

[0034] The inference engine 73A uses the inference model recorded in the inference model recording unit 72A to infer image data input to it via the system control unit 50A. As the inference model, it can use an inference model input from an external source (e.g., external device 101) via the communication unit 71A and recorded in the inference model recording unit 72A, or an inference model obtained by retraining using the training unit 74A. Note that if the inference model is updated externally and then retrained using the training unit 74A, the managed version of the updated inference model is maintained in the inference model recording unit 72A, etc., making it possible to identify each inference model. Furthermore, the inference engine 73A has a neural network design 73a.

[0035] In neural network design 73a, intermediate layers (neurons) are arranged between the input layer and the output layer. Image data is input to the input layer from the system control unit 50A. Several layers of neurons are arranged as intermediate layers. The number of neuron layers is appropriately set from a design perspective, and the number of neurons in each layer is also appropriately set from a design perspective. The intermediate layers are weighted based on the inference model recorded in the inference model recording unit 72A. Annotation information corresponding to the image data input to the input layer is output to the output layer.

[0036] In this embodiment, an inference model is envisioned for inferring the classification of subjects included in an image. As the inference model, an inference model has been generated by deep learning by an external device 101, etc., in which image data of various subjects and their classification results (e.g., animal classification such as "dog" and "cat", subject classification such as "human", "animal", "plant", and "building") are used as training data. However, since the inference engine 73A can be updated from the external device 101, etc., there will be no problem in dealing with various other inference models.

[0037] The training unit 74A retrains the inference model upon receiving a request from the system control unit 50A, etc. The training unit 74A includes a training data recording unit 74a for recording information related to the training data to be input into the inference engine 73A. The training unit 74A can use the training data recorded in the training data recording unit 74a to retrain the inference engine 73A, and uses the inference model recording unit 72A to update the inference engine 73A.

[0038] Communication unit 71A includes communication circuitry for transmitting and receiving data. Specifically, the communication circuitry can perform wireless communication such as Wi-Fi or Bluetooth (registered trademark), or wired communication such as Ethernet or USB. Communication unit 71A can communicate with communication unit 71B of external device 101 and communication unit 71C of external display device 102. Communication unit 71A functions as a communication unit that not only transmits and receives image files with annotation information created by inference engine 73A relative to system control units 50A, 50B, and 50C, but also transmits and receives various types of information such as inference models and training data relative to system control units 50A, 50B, and 50C. Furthermore, communication unit 71A can impose restrictions on the information transmitted based on whether external device 101 and external display device 102 are associated with digital camera 100.

[0039] External device 101 includes a training unit 74B, an inference engine 73B, an inference model recording unit 72B, a system control unit 50B, and a communication unit 71B. Alternatively, external device 101 may be a device without such components. The training unit 74B creates an inference model upon receiving a request from an external source (e.g., the inference engine 73B or the system control unit 50B). The inference model recording unit 72B records the inference model transmitted from the digital camera 100 and the inference model created by the training unit 74B.

[0040] Figure 1B An example of a detailed configuration of an external display device 102 is shown. The external display device 102 includes an image processing unit 20C, a memory control unit 22C, a display unit 23C, an image display memory 24C, a memory 25C, a non-volatile memory 51C, an operation unit 62C, a communication unit 71C, a metadata analysis unit 75, a power supply 80C, and a power control unit 81C. The external display device 102 can receive image files, etc., from the digital camera 100 via the communication unit 71C, and display the image files, etc., received by the system control unit 50C on the display unit 23C.

[0041] Next, refer to Figure 2 The image processing of the digital camera 100 in this embodiment is described. For example, the processing according to this flowchart can be implemented by the system control unit 50A executing the corresponding program (stored in the non-volatile memory 51, etc.) and controlling the function blocks of the digital camera 100.

[0042] Processing begins when an instruction indicating the camera mode is provided using the mode switch included in the operation unit 63. In step S201, it is checked whether an instruction to capture a still image has been provided by the user pressing the shutter button 60 and thus activating the shutter switches 61 and 62 (SW1 and SW2), and if an instruction to capture a still image has been provided, the process proceeds to step S202.

[0043] Image processing is performed in step S202. During image processing, autofocus (AF) and auto exposure (AE) processing are performed using the focus control unit 41 and exposure control unit 40, and then the image signal output from the image sensor 13 via the A / D converter 15 is stored in the memory 25. Furthermore, image data is created, wherein the image signal stored in the memory 25 is compressed in JPEG or MPEG-4 HEVC format using compression processing included in the image processing unit 20 according to user settings.

[0044] In step S203, the image processing unit 20 performs subject detection processing on the image signal stored in the memory 25 and obtains the detection information of the subject included in the image.

[0045] In step S204, the image data created in step S202 and the subject detection information obtained in step S203 are recorded as image files on the external recording medium 91. The recorded image files are formatted as follows: Figure 4A The format shown is used for recording. In this embodiment, the recorded image file 400 consists of at least an area storing metadata conforming to the Exif standard and an image data area storing compressed image data 406. For example, if the user has provided an instruction to record in JPEG format, the image file 400 is recorded in JPEG format, and the Exif data 401 is recorded in the APP1 tag, etc. Furthermore, if the user has provided an instruction to record in High-Efficiency Image File Format (HEIF) format, the image file 400 is recorded in HEIF file format, and the Exif data 401 is recorded in the metadata box, etc. Similarly, if the user has provided an instruction to record in RAW format, the Exif data 401 is recorded in a predetermined area such as the metadata box.

[0046] Using the metadata generation and analysis unit 70, the subject detection information obtained in step S203 is stored in a MakerNote 404 included in the Exif data 401. The manufacturer (or supplier, hereinafter the same) of the digital camera 100 may record the MakerNote 404 in its own proprietary recording format, and because its internal specifications and format are not disclosed in the Exif standard, etc., only the manufacturer can understand the contents of the MakerNote 404. Furthermore, the MakerNote 404 is an area that can only be used with actual software produced by the supplier (e.g., applications installed on personal computers, tablets, or smartphones, or web services accessible from these terminals, etc.). The MakerNote 404 includes annotation information 405. Additionally, the subject detection information is recorded in annotation information 403, which is recorded at the position indicated by the annotation information offset included in the annotation link information storage label 402. The subject detection information recorded in annotation information 403 is used as annotation information input to the inference engine 73A as input data in the inference process described later. Figure 4A In the example shown, only the coordinates of the subject within the frame are recorded in annotation information 403. However, other information may also be included.

[0047] In step S205, it is checked whether inference processing using the inference engine 73A can be performed. For example, inference processing cannot be performed when the inference engine 73A is currently processing another image, or when an instruction to shoot in continuous shooting mode is given by operating the shutter button 60 and the shutter switch 62 (SW2) is therefore kept open (in which case shooting processing takes precedence over inference processing). If inference processing cannot be performed, the process proceeds to step S209, and if inference processing can be performed, the process proceeds to step S206.

[0048] In step S206, the image file 400 undergoes inference processing using the inference engine 73A. In this embodiment, the image file 400 is input to the inference engine 73A. Subject regions included in the image data are specified based on the annotation information 403 and image data 406 included in the image file 400, and the classification results of the subjects included in the subject regions are output as the results of inference using the inference engine 73A for each subject region. During inference, in addition to the inference results, information related to the inference processing, such as logs and debugging information related to operations performed during inference, may also be output.

[0049] In step S207, the subject classification result (i.e., the inference result obtained in step S206) is classified into a first inference result and a second inference result as an inference result other than the first inference result according to a predetermined criterion. The first inference result is recorded in the image file, and the process proceeds to step S208. Figure 4B The representative inference results for each subject have been recorded as the first inference result. Figure 4A Image file 410 is subsequently obtained from image file 400. Using metadata generation and analysis unit 70, in image file 410, representative inference results of representative subjects are recorded in annotation information 411, and representative inference results of each subject including the main subject are recorded in annotation information 412 in MakerNote 404.

[0050] Here, for example, as a predetermined criterion mentioned above, one of the following criteria may be adopted: whether the inference result has the highest reliability; and whether the inference result is specific to the main subject. The details of the predetermined criteria will be described in detail below.

[0051] First, regarding reliability, the image processing unit 20 can determine the reliability of the inference result based at least in part on at least one of a plurality of imaging conditions including: the size of the subject area; the blur of the subject area; whether the subject area is hidden or partially cut off by other subject areas; and the exposure and brightness when the subject is captured. For example, the reliability can have a maximum value of 100 and a minimum value of 0, respectively. Specifically, if the subject area is an image captured under imaging conditions suitable for subject classification (e.g., if the subject area is an image captured with appropriate exposure and brightness without any blur and without being hidden or cut off by other subjects), the reliability of the inference result will be high. On the other hand, if the subject area is an image captured under imaging conditions unsuitable for subject classification (e.g., if the subject area is an image captured in a blurred state or hidden or cut off by other subjects, or an image captured under backlight conditions or in a very dark environment), the reliability of the inference result will be low.

[0052] Next, regarding the main subject, the main subject and other subjects can be determined based on at least one of the following: the size of the subject area in the image data, the size of a specific portion of the subject in the image data, and the amount of displacement relative to the center of the image. Specifically, the main subject can be the subject corresponding to the largest subject area in the image data, the subject whose specific portion, such as a face, is the largest in the image data, or the subject with the smallest displacement relative to the center of the image data (or in other words, the subject closest to the center). Furthermore, subjects detected within an area specified by the user of the digital camera 100 can be determined as the main subject.

[0053] exist Figure 4B In the example shown, multiple subjects, including at least subjects A and B, are detected from image data 406. Among the subjects, subject A is identified as the primary subject, and the representative inference result of subject A is recorded in annotation information 411. Furthermore, the representative inference results of each of the multiple subjects detected from image data 406 are recorded in MakerNote 404. In this way, the inference processing results constituting know-how and assets can be handled as follows: while some information (whose description format is already disclosed in standards such as Exif) can be stored in areas that can be shared between manufacturers, most information (whose description format is not disclosed in standards such as Exif) can be maintained and protected in areas specific to the manufacturer.

[0054] Next, in step S208, the inference results not classified as the first inference result (i.e., the second inference result) are recorded in a management file, which is a file on the external recording medium 91 that is different from the image files. If the current inference model's management version, debugging information, etc., are maintained in the inference model recording unit 72A, such information is recorded in the management file as inference model management information. Although the image files are configured as independent files for each image, the management file can be configured such that information related to multiple image files is included in a single file.

[0055] Since the management file is not a file type created every time image data 406 is generated, it can be created in advance on the external recording medium 91, and information corresponding to the image file can be added to the management file each time an image file is created. This allows management files to be created independently for each user of the digital camera 100.

[0056] Furthermore, management files can be recorded in a manufacturer-specific format, and management files can be created whose content is only understandable to the manufacturer because its internal specifications and format are not disclosed in standards such as DCF or Exif. Additionally, management files can be configured to be used only with pre-defined physical software produced by the manufacturer (e.g., applications installed on personal computers, tablets, or smartphones, or web services accessible from these devices), and can be referenced by that pre-defined physical software.

[0057] In the subsequent step S209, the recording process ends once the user provides an instruction to complete the recording by using the mode switch included in the operating unit 63, which is being operated, or as a result of turning off the power by using the power button included in the operating unit 63.

[0058] Figures 4A to 4C This is a diagram illustrating an example of data configuration for image files and management files according to this embodiment. Figure 4A An example of the configuration of an image file is shown in which the image data created in step S202 and the subject detection information obtained in step S203 are recorded as an image file on an external recording medium 91 in step S204. Figure 4B This shows that some inference results have been recorded in Figure 4A An example of the configuration of image file 410 obtained later from image file 400. Figure 4C An example configuration of management file 420, which records inference results, inference model management information, etc., is shown.

[0059] exist Figure 4BIn the image file 410, using the metadata generation and analysis unit 70, representative inference results of the main subject are associated with the position coordinates of the subject and recorded in annotation information 411. Since annotation information 411 is recorded in a format defined in the Exif standard, it can be used regardless of the manufacturer of the digital camera 100. Furthermore, as annotation information 412 in MakerNote 404, representative inference results of the main subject and other subjects are recorded in a manner that associates each with the position coordinates of the subject. In this embodiment, the inference results recorded in the image file 410 are collectively referred to as first inference results.

[0060] Figure 4C The management file 420 shown stores multiple management information entries 421A to 421N separately for n image files (where n is a natural number of 1, 2, or greater). Here, management information 421A indicates... Figure 4B The image file 410 shown corresponds to the management information. Although the image files corresponding to other pieces of management information (i.e., management information 421B, etc.) also exist in a similar format, in... Figure 4B Such image files are omitted. Multiple management information entries 421A to 421N record multiple image information entries 422A to 422N for specifying the corresponding image file 410. For example, image file number, recording date and time, image format, etc., can be included in image information 422, and the corresponding image file 410 can be specified based on the image file number in this information. Furthermore, Figure 4C The following scenario is shown as an example, in which, in each of the annotation information entries 423A to 423N, all inference results, including the first inference result recorded in the image file 410, are recorded for each subject in a manner associated with the position coordinates of each subject. Furthermore, inference model management information 424 is also recorded in each of the management information entries 421.

[0061] In this embodiment, an example is described where the annotation information 411, 412, and 423 are composed of the free position coordinates and the inference results. However, it is sufficient as long as the output of the inference model can be recorded in the annotation information. For example, the output results can be directly recorded in the annotation information, and the recording format or form is not limited; that is, the output results can be recorded in text form or binary form, etc.

[0062] In this way, image data 406 and some of the inference results (e.g., representative inference results among the multiple inference results) are recorded in image file 410. Therefore, once recording is complete, image data 406 and inference results can be effectively correlated and easily managed, without needing to be managed separately. Furthermore, the process for generating image file 410 can be simplified by recording only representative inference results, and the file size can also be reduced. Additionally, inference model management information 424, which is proprietary data used by the manufacturer managing the inference model, can be associated with the inference results and securely managed by not being widely disclosed to the public and being recorded in an unpublished management file 420.

[0063] In the above description, representative inference results are recorded in the image file, and all inference results are recorded in the management file. However, when using whether the inference result applies to the main subject as a predetermined criterion, all inference results for the main subject can be recorded in the image file, and all inference results for all subjects can be recorded in the management file. In this case, it is sufficient that the representative inference result for the main subject is recorded in annotation information 411 and all inference results for the main subject are recorded in annotation information 412 in the image file. It is sufficient to determine the representative inference result in this case based on whether the inference result has the highest reliability.

[0064] Furthermore, in the annotation information 412 in the image file and the annotation information 423 in the management file, only the differences from annotation information 411 and annotation information 412 can be recorded, instead of recording all inference results. Additionally, instead of judging based on the reliability of the inference results or whether the inference results pertain to the main subject, predetermined items in the inference results can be used as representative inference results. For example, representative inference results can be predetermined as information indicating whether the subject is a person, whether the subject is an animal, etc., and other inference results can be information such as the subject's personal name, gender, and age.

[0065] Next, refer to Figure 3 The playback process of the digital camera 100 according to this embodiment is described. For example, the process according to this flowchart can be implemented by the system control unit 50A executing the corresponding program (stored in the non-volatile memory 51, etc.) and controlling the function blocks of the digital camera 100.

[0066] In the digital camera 100, when an instruction to indicate a playback mode is provided using a mode switch included in the operation unit 63, playback processing for viewing image data recorded on the external recording medium 91 begins. Once playback processing begins, in step S301, an image file to be played back is specified and loaded from the external recording medium 91 into the memory 25 via the card controller 90.

[0067] In step S302, the image processing unit 20 decompresses and resizes the image file loaded into the memory 25 according to the recording format, converts it into display image data, and records the display image data in the image display memory 24 and displays it by the display unit 23.

[0068] In step S303, the metadata generation and analysis unit 70 checks whether the inference result of the currently playing image file has been recorded in the management file 420. If the inference result has been recorded, the process proceeds to step S304; otherwise, the process proceeds to step S307. In step S304, if the user provides an instruction to display the inference result already recorded in the image file using the operation component included in the operation unit 63, the process proceeds to step S305. Otherwise, the process proceeds to step S310.

[0069] In step S305, the metadata generation and analysis unit 70 analyzes the management file 420, specifies the management information of the currently playing image file from multiple management information 421, and obtains annotation information 423. The process then proceeds to step S306. In step S306, the metadata generation and analysis unit 70 extracts information related to the inference results from the annotation information 423 and displays the extracted information using the image processing unit 20, overlaying it onto the currently playing image data. For example, the inference results associated with the image data are displayed in a user-visible manner by showing strings indicating the inference results obtained by classifying the subjects within the coordinate regions, along with boxes indicating the coordinate regions of the subjects described in the annotation information 423. The process then proceeds to step S310.

[0070] In step S307, a process similar to that in step S206 is performed, and the process proceeds to step S308. Similar to step S207, in step S308, the subject classification result (i.e., the inference result obtained in step S307) is classified as a first inference result and a second inference result as an inference result other than the first inference result. The first inference result is recorded in image file 410, and the process proceeds to step S309. Similar to step S208, in step S309, the second inference result (i.e., the inference result not classified as the first inference result), image information, inference model management information, etc., are recorded in management file 420, which is a file different from image file 410. Here, the first inference result is also recorded in the management file. Then, the process continues to step S310.

[0071] In step S310, if an instruction to play back other images is provided using the operation member included in the operation unit 63, the process returns to step S301, and the above process is repeated for the image to be played back as indicated by the instruction. On the other hand, if no instruction to play back other images is provided, in step S311 it is determined whether an instruction to terminate the playback process has been provided, and the process returns to step S303. If no instruction to terminate the playback process is provided, the above process is repeated; if an instruction to terminate the playback process is provided, the playback process is terminated.

[0072] In this embodiment, by recording only the first inference result (i.e., the representative inference result) in image file 410, the processing performed by the metadata generation and analysis unit 70 to display the inference result is reduced, and the processing speed can be improved. If detailed inference results are required, management file 420 can be used. Since management information 421 for multiple image files is recorded in management file 420, the metadata generation and analysis unit 70 can obtain and use the inference results of multiple image files at once by analyzing one management file 420.

[0073] Next, refer to Figure 5 The transmission process of the digital camera 100 in this embodiment will be described. Here, the description will be provided as an example of recording the image file before transmission in the format of the image file 410 described above, and also recording the management file 420.

[0074] When a transmission command is provided using the operating component included in the operating unit 63, the process begins. Figure 5 The sending process is shown in the flowchart. Sending instructions can be provided as follows: sending can begin as a result of selecting one or more image files as objects; pre-settings can be made to automatically send images when they are captured; and sending can be triggered by the generation of image files. Here, it will be described in accordance with the above... Figure 4BThe image file 410 in the image file records the image file before it is sent and follows the format described above. Figure 4C The following description uses the format of management file 420 as an example to illustrate the management file situation. For example, according to... Figure 5 The processing of the flowchart can be achieved by the system control unit 50A executing the corresponding program (stored in the non-volatile memory 51, etc.) and controlling the function blocks of the digital camera 100.

[0075] exist Figure 5 In step S501, the digital camera 100 requests system information from the external display device 102 via its communication unit 71A, and the external display device 102 transmits the system information from the system control unit 50C to the digital camera 100 via its communication unit 71C. Therefore, the digital camera 100 can acquire the system information from the external display device 102. Then, the process proceeds to step S502.

[0076] In step S502, the system control unit 50A determines the method of transmission to the external display device 102. Specific examples of transmission methods include wireless transmission such as Wi-Fi or Bluetooth (registered trademark), wired transmission via Ethernet cable or USB cable, and transmission using removable media such as an SD card. As a method for determining the transmission method to be used, if multiple transmission methods are available, the transmission method set by the user using the operating member included in the operation unit 63 can be used, or the transmission method to be used can be determined based on the connection status between the external device 101 and the digital camera 100. If only one transmission method is available, that transmission method can be determined as the transmission method to be used.

[0077] In the subsequent step S503, the system control unit 50A determines whether the external display device 102 can use the management file 420 based on the system information of the external display device 102 obtained in step S501. As described above, since the management file 420 is recorded in a manufacturer-specific description format not disclosed in standards such as Exif, it can be referenced using only a specific application or web service. Here, it is assumed that the external display device 102 is a tablet terminal, and an application provided by the manufacturer of the digital camera 100 or an application compatible with the manufacturer's application is installed on the tablet terminal. In this case, the digital camera 100 and the external display device 102 are connected using the application, and the system information provided from the external display device 102 includes information indicating that the application has been installed. Since the management file 420 can be referenced by using the application, if the application has been installed on the external display device 102, it can be determined that the external display device 102 can use (reference) the management file 420.

[0078] In addition to the above, the ability of the external display device 102 to use the management file 420 can be determined based on whether a response is returned when a predetermined communication command (e.g., a manufacturer-specific communication command) is sent from the digital camera 100 to the external display device 102. Furthermore, based on received system information, the determination can be made based on information such as whether user registration has been completed or whether fees have been paid. Regarding user registration, it can be determined that the external display device 102 can only use the management file 420 if a comparison with the user information of the digital camera 100 is performed and a match is found. This is beneficial for protecting the management file 420 in cases such as when a third party accesses the digital camera 100 using an application provided by the manufacturer to maliciously use the digital camera 100.

[0079] If it is determined that the external display device 102 can use the management file 420, the process proceeds to step S504, and the sending process of the management file 420 is executed. On the other hand, if the external display device 102 cannot use the management file 420, the sending process is skipped, and the process proceeds to step S505.

[0080] In the subsequent step S504, the system control unit 50A uses the method determined in step S502 for sending to the external display device 102 to perform object management file transmission processing via the communication unit 71A, and then the process proceeds to step S505. The external display device 102 can display the inference results on the display unit 23C by analyzing the management file received from the communication unit 71C using the metadata analysis unit 75 in a manner similar to that in step S305. In step S505, one or more object image files are transmitted using the transmission method determined in step S502, and the process terminates.

[0081] In this way, according to the above-described sending process, by sending only management files to specific destinations, it is possible to prevent know-how and assets (data created at great expense) from leaking to unspecified destinations.

[0082] Other embodiments

[0083] Note that the present invention can be applied to systems consisting of multiple devices (e.g., host computer, interface device, scanner, camera, etc.) or devices consisting of a single device (e.g., copier, fax machine, etc.).

[0084] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0085] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: A detection unit is configured to detect multiple subjects, including a main subject and other subjects, from an image; A processing unit is configured to perform inference processing for classifying a plurality of detected subjects to obtain an inference result including a first inference result and a second inference result, wherein the first inference result is at least a portion of the inference result of the inference processing for the main subject among the plurality of detected subjects, and the second inference result includes the inference result of the inference processing for the other subjects among the plurality of detected subjects. A generation unit configured to generate an image file including image data of the image and a first inference result for the main subject but excluding a second inference result for the other subjects, and to record the image file in a recording medium, wherein the first inference result is included in the image file in a first format defined in a predetermined standard; and A storage unit configured to store management information, including information for specifying the image file and management information for the second inference result for the other subject, in a management file different from the image file and recorded on the recording medium, wherein the management file includes management information for a plurality of the images in a second format specific to the manufacturer of the image processing device, and wherein the second format does not allow the use of the management file without the use of software from the manufacturer.

2. The image processing apparatus according to claim 1, wherein, The management information also includes information related to the inference model used by the processing unit for the inference process.

3. The image processing apparatus according to claim 1, wherein, The main subject is determined from a plurality of detected subjects based on at least one of the following: the size of the subject region in the image, the size of a specific portion of the subject in the image, and the amount of displacement of the subject relative to the center of the image.

4. The image processing apparatus according to claim 1, wherein, The main subject is the subject detected within a designated area in the image.

5. The image processing apparatus according to any one of claims 1 to 3, wherein, The management information also includes the first inference result.

6. The image processing apparatus according to any one of claims 1 to 3, wherein, The management file includes management information associated with multiple image files.

7. The image processing apparatus according to any one of claims 1 to 3, further comprising: A determination unit is configured to determine whether an external display device connected to the image processing device can use the management file; as well as A sending unit configured to send the image file to the external display device. Wherein, if the determining unit determines that the external display device can use the management file, the sending unit sends the image file and the management file to the external display device, and If the determination unit determines that the external display device cannot use the management file, the sending unit only sends the image file to the external display device.

8. The image processing apparatus according to claim 7, wherein, When the external display device uses predetermined software that can be used to reference the management file, the determination unit determines that the external display device can use the management file.

9. The image processing apparatus according to claim 7, wherein, If the external display device returns a response to the predetermined command, the determination unit determines that the external display device is capable of using the management file.

10. The image processing apparatus according to claim 7, wherein, If a predetermined user registration has already been performed on the external display device, the determination unit determines that the external display device is capable of using the management file.

11. The image processing apparatus according to any one of claims 1 to 3, further comprising a camera unit configured to capture the image.

12. A control method for an image processing device, the control method comprising: Detect multiple subjects in an image, including the main subject and other subjects; An inference process is performed to classify multiple detected subjects to obtain an inference result including a first inference result and a second inference result, wherein the first inference result is at least a portion of the inference result of the inference process for the main subject among the multiple detected subjects, and the second inference result includes the inference result of the inference process for the other subjects among the multiple detected subjects; Generate an image file including image data of the image and a first inference result for the main subject but excluding a second inference result for the other subjects, and record the image file in a recording medium, wherein the first inference result is included in the image file in a first format defined in a predetermined standard; and Management information, including information for specifying the image file and the second inference result for the other subjects, is stored in a management file that is different from the image file and recorded on the recording medium. The management file includes management information for a plurality of the images in a second format specific to the manufacturer of the image processing device, and the second format does not allow the use of the management file without the use of software from the manufacturer.

13. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by a processor of an image processing device, implement the control method according to claim 12.

14. A computer program product comprising one or more programs, said one or more programs including instructions that, when executed by a processor of an image processing device, implement the control method according to claim 12.

Images