Image processing device and method, and imaging device

Abstract

To properly manage an estimation result of a subject in an image.SOLUTION: An image processing apparatus comprises: detection means for detecting a subject from an image; inference means for performing inference processing of the detected subject by using an inference model; and generation means which collects image data of the image, information of the subject, an inference result of the inference processing, and information of the inference model and generates an image file. The generation means records the inference model in a non-disclosed area of the image file.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device and method for processing images captured by a digital camera or the like, and to an imaging device. [Background technology]

[0002] In recent years, artificial intelligence (AI) technologies such as deep learning have been utilized in various technical fields. For example, conventionally, digital still cameras and the like have been equipped with a function for detecting human faces from captured images, but Patent Document 1 discloses a technology for not only detecting people, but also accurately detecting and recognizing animals such as dogs and cats.

[0003] On the other hand, as subject detection technology evolves and the importance of the information that can be detected increases, the input image as the detection target and the detection results are becoming important information that can be utilized in various ways in the subsequent workflow. For example, various information estimated from images using AI technology is becoming very important input data in various fields such as robot automation and self-driving cars. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-99559 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is no mention of a method for managing the information detected from the input image.

[0006] The present invention has been made in consideration of the above problems, and has an object to appropriately manage the estimation results of a subject in an image. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the image processing device of the present invention has a detection means for detecting a subject from an image, an inference means for performing inference processing on the detected subject using an inference model, and a generation means for generating an image file by combining image data of the image, information about the subject, the inference result of the inference processing, and information about the inference model, and the generation means records the inference model in a private area of ​​the image file. Effect of the Invention

[0008] According to the present invention, it is possible to appropriately manage the estimation results of the subject in an image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a digital camera according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart showing a photographing process of the digital camera according to the embodiment. [Diagram 3] 4 is a flowchart showing a playback process of the digital camera according to the embodiment. [Figure 4] 6 is a flowchart showing a re-inference process of the digital camera according to the embodiment. [Diagram 5] 2A and 2B are diagrams illustrating an overview of the recorded contents of an image file recorded by a digital camera according to the embodiment. [Figure 6] 5 is a flowchart showing a transmission process of the digital camera according to the embodiment. [Figure 7] 5A and 5B are diagrams illustrating an overview of the recorded contents of an image file during a transmission process of the digital camera according to the embodiment. [Figure 8] 4 is a flowchart showing an editing process of the digital camera according to the embodiment. [Figure 9] 4A and 4B are diagrams illustrating an overview of the recorded contents of an image file during editing processing of the digital camera according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the following description, a digital camera is exemplified as an image output device that uses an inference model to classify objects, but it goes without saying that the image output device of the present invention is not limited to a digital camera. The image output device of the present invention may be any device that reads out images recorded in a recording device and displays them on a display device, and may be, for example, a smartphone, a tablet PC, or the like.

[0012] FIG. 1 is a block diagram showing an example of the configuration of a digital camera 100 according to an embodiment of the present invention. The barrier 10 is a protective member that covers the imaging section including the photographing lens 11 of the digital camera 100 to prevent the imaging section from becoming dirty or damaged, and its operation is controlled by a barrier control section 43. The photographing lens 11 forms an optical image on the imaging surface of the imaging element 13. The shutter 12 has an aperture function. The imaging element 13 is composed of, for example, a CCD or CMOS sensor, and converts the optical image formed on the imaging surface by the photographing lens 11 via the shutter 12 into an electrical signal.

[0013] The A / D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal converted by the A / D converter 15 is written to the memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on information at the time of shooting and written to the memory 25. The development parameters are made up of various parameters used in image processing for recording in the JPEG format, such as exposure settings, white balance, color space, and contrast.

[0014] The timing generator 14 is controlled by the memory control unit 22 and the system control unit 50 A, and supplies clock signals and control signals to the image sensor 13 , the A / D converter 15 , and the D / A converter 21 .

[0015] The image processing unit 20 performs various image processing such as predetermined pixel interpolation processing, color conversion processing, correction processing, and resizing processing on the data from the A / D converter 15 or the data from the memory control unit 22. The image processing unit 20 also performs predetermined image processing and calculation processing using image data obtained by capturing an image, and provides the obtained calculation results to the system control unit 50A. The system control unit 50A realizes AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing by controlling the exposure control unit 40 and the focus control unit 41 based on the provided calculation results.

[0016] The image processing unit 20 also performs a predetermined calculation process using the captured image data, and performs AWB (auto white balance) processing based on the calculation results. Furthermore, the image processing unit 20 reads image data stored in the memory 25 and performs compression or decompression processing such as JPEG, MPEG-4 AVC, HEVC (High Efficiency Video Coding), or lossless compression of uncompressed RAW data. The image processing unit 20 then writes the processed image data to the memory 25.

[0017] The image processing unit 20 also performs a predetermined calculation process using the captured image data, and performs editing process of various image data. Specifically, it can perform a trimming process to adjust the display range and size of the image by hiding unnecessary parts around the image data, and a resizing process to change the size of the image data and the display elements on the screen by enlarging or reducing them. Furthermore, it can perform RAW development to add image processing such as color conversion to data that has been compressed or expanded, such as lossless compression of uncompressed RAW data, and convert it to the JPEG format to create image data. It can also perform video clipping process to clip out a specified frame of a video format such as MPEG-4, convert it to the JPEG format, and save it.

[0018] The image processing unit 20 also performs processing such as superimposing an OSD (On-Screen Display) such as a menu or arbitrary characters to be displayed on the display unit 23 together with the image data for display. Furthermore, the image processing unit 20 performs subject detection processing to detect the subject present in the image data and detect the subject area using the input image data and distance information from the subject obtained from the image sensor 13 at the time of shooting. Detectable information includes area such as position and size in the image, inclination, likelihood, and other detection information.

[0019] The memory control unit 22 controls the A / D converter 15, the timing generator 14, the image processing unit 20, the image display memory 24, the D / A converter 21, and the memory 25. The RAW image data generated by the A / D converter 15 is written into the image display memory 24 or the memory 25 via the image processing unit 20 and the memory control unit 22, or directly via the memory control unit 22.

[0020] The image data for display written in the image display memory 24 is displayed on a display unit 23 configured with a TFT LCD or the like via a D / A converter 21. By sequentially displaying image data obtained by capturing an image on the display unit 23, it is possible to realize an electronic finder function that displays a live image. The memory 25 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of video, and stores the captured still images and video. The memory 25 can also be used as a working area for the system control unit 50A.

[0021] The exposure control unit 40 controls the shutter 12 having an aperture function. The exposure control unit 40 also has a flash dimming function by linking with the flash 44. The focus control unit 41 adjusts the focus by driving a focus lens (not shown) included in the photographing lens 11 based on instructions from the system control unit 50A. The zoom control unit 42 controls zooming by driving a zoom lens (not shown) included in the photographing lens 11. The flash 44 has an AF assist light projection function and a flash dimming function.

[0022] The system control unit 50A controls the entire digital camera 100. The non-volatile memory 51 is an electrically erasable and recordable non-volatile memory, such as an EEPROM. Note that the non-volatile memory 51 stores not only programs but also map information and the like.

[0023] The shutter switch 61 (SW1) is turned on during the operation of the shutter button 60, and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) is turned on when the operation of the shutter button 60 is completed, and instructs the start of a series of shooting operations including exposure processing, development processing, and recording processing. In the exposure processing, a signal read from the image sensor 13 is written to the memory 25 as RAW image data via the A / D converter 15 and the memory control unit 22. In the development processing, the RAW image data written to the memory 25 is developed using calculations in the image processing unit 20 and the memory control unit 22, and written to the memory 25 as image data. In the recording processing, the image data is read from the memory 25, compressed by the image processing unit 20, and the compressed image data is stored in the memory 25 and then written to the external recording medium 91 via the card controller 90.

[0024] The operation unit 63 includes various operation members such as buttons and a touch panel. For example, it includes a power button, a menu button, a mode change switch for changing between shooting mode, playback mode, and other special shooting modes, a cross key, a set button, a macro button, and a multi-screen playback page change button. It also includes, for example, a flash setting button, a single shot / continuous shooting / self-timer switching button, a menu movement + (plus) button, a menu movement - (minus) button, a shooting image quality selection button, an exposure compensation button, and a date / time setting button.

[0025] When recording image data on the external recording medium 91, the metadata generating and analyzing unit 70 generates various metadata such as Exif (Exchangeable image file format) standard to be attached to the image data based on information at the time of shooting. In addition, when reading image data recorded on the external recording medium 91, the metadata generating and analyzing unit 70 analyzes the metadata added to the image data. Examples of metadata include shooting setting information at the time of shooting, image data information related to the image data, and feature information of the subject included in the image data. In addition, when recording moving image data, the metadata generating and analyzing unit 70 can generate and add metadata for each frame.

[0026] The power supply 80 is composed of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, etc. The power supply control unit 81 supplies the power from the power supply 80 to each unit of the digital camera 100. The card controller 90 transmits and receives data to and from an external recording medium 91, such as a memory card. The external recording medium 91 is formed, for example, of a memory card, and records images (still images and videos) captured by the digital camera 100.

[0027] The inference engine 73A uses the inference model recorded in the inference model recording unit 72A to perform inference on image data input via the system control unit 50A. The inference model may be an inference model input from the outside, such as the external device 101, via the communication unit 71A and recorded in the inference model recording unit 72A, or an inference model obtained by re-learning by the learning unit 74A. Note that the inference model is assumed to hold a management version in the inference model recording unit 72A or the like so that each inference model can be identified when it is updated from the outside or updated by the learning unit 74A through re-learning. The inference engine 73A also has a neural network design 73a.

[0028] In the neural network design 73a, an intermediate layer (neurons) is arranged between the input layer and the output layer. Image data is input to the input layer from the system control unit 50A. Several layers of neurons are arranged as intermediate layers. The number of neuron layers is appropriately determined in design, and the number of neurons in each layer is also appropriately determined in design. The intermediate layers are weighted based on the inference model recorded in the inference model recording unit 72A. Annotation information according to the image data input to the input layer is output to the output layer.

[0029] In this embodiment, an inference model that infers the classification of a subject included in an image is assumed. Image data of various subjects and the classification results (for example, classification of animals such as dogs and cats, and classification of subjects such as people, animals, plants, buildings, etc.) are used as teacher data, and an inference model generated by deep learning using an external device 101 or the like is used. However, since the inference engine 73A can be updated from the external device 101 or the like, various other inference models may be used.

[0030] The learning unit 74A re-learns the inference model upon request from the system control unit 50A etc. The learning unit 74A has a teacher data recording unit 74a, which records information related to teacher data for the inference engine 73A. The learning unit 74A re-learns the inference engine 73A using the teacher data recorded in the teacher data recording unit 74a, and can update the inference engine 73A using the inference model recording unit 72A.

[0031] The communication unit 71A has a communication circuit for transmitting and receiving. Specifically, the communication circuit may be wireless communication such as Wi-Fi or Bluetooth (registered trademark), or may be wired communication such as Ethernet or USB. The communication unit 71A can communicate with the communication unit 71B of the external device 101. The communication unit 71A functions as a communication unit that transmits and receives not only image files to which annotation information created in the inference engine 73A is added, but also various information such as inference models and teacher data to the system control unit 50A and the system control unit 50B. In addition, the information to be transmitted can be restricted depending on whether the external device 101 is associated with a camera.

[0032] The external device 101 has a learning unit 74B, an inference engine 73B, an inference model recording unit 72B, a system control unit 50B, and a communication unit 71B. Alternatively, the external device 101 may be a device that does not have these configurations. The learning unit 74B creates an inference model upon request from an external source, such as the inference engine 73B or the system control unit 50B. The inference model recording unit 72B records the inference model transmitted from the digital camera 100 and the inference model created by the learning unit 74B.

[0033] Next, the photographing process of the digital camera 100 according to this embodiment will be described with reference to FIG. The process starts when the shooting mode is instructed by the mode change switch included in the operation unit 63. In S201, it is confirmed whether the user has pressed the shutter button 60 to turn on the shutter switches 61 (SW1) and 62 (SW2) to instruct still image shooting, and if still image shooting is instructed, the process proceeds to S202.

[0034] In S202, an image capturing process is performed. In the image capturing process, AF (autofocus) process and AE (autoexposure) process are performed using the focus control unit 41 and the exposure control unit 40, and then the image signal output from the image sensor 13 via the A / D converter 15 is stored in the memory 25. Furthermore, image data is created by compressing the image signal stored in the memory 25 in the JPEG format or the MPEG-4 HEVC format using a compression process included in the image processing unit 20 in accordance with the user's settings.

[0035] In S203, the image processing unit 20 performs subject detection processing on the image signal stored in the memory 25, and obtains detection information of the subject included in the image.

[0036] In S204, the image data created in S202 and the subject detection information acquired in S203 are recorded as an image file in the external recording medium 91. The image file recorded here is recorded in a format as shown in FIG. 5(a). The image file 400 recorded in this embodiment is composed of an area for storing at least metadata conforming to the Exif standard and an image data area for recording compressed image data 406. For example, if the user instructs recording in the JPEG format, the image file 400 is recorded in the JPEG format, and the Exif data 401 is recorded in the APP1 marker or the like. If the user instructs recording in the HEIF (High Efficiency Image File Format) format, the image file 400 is recorded in the HEIF file format, and the Exif data 401 is recorded in the Metadata Box or the like. Similarly, if recording in the RAW format is instructed, the Exif data 401 is recorded in a predetermined area such as the Metadata Box.

[0037] The subject detection information acquired in S203 is stored in MakerNote 404, in which metadata specific to the manufacturer included in the Exif data 401 can be written in a format that is generally not publicly disclosed, by using the metadata generating and analyzing unit 70. Furthermore, the subject detection information is recorded in annotation information 403a recorded at a position indicated by an annotation information offset included in the annotation link information storage tag 402. The subject detection information recorded in annotation information 403a is used as annotation information that becomes input data to inference engine 73A in the inference process described later. In the example shown in FIG. 5(a), only the coordinate area of ​​the subject within the screen is recorded in annotation information 403a, but other information may also be included.

[0038] In S205, it is confirmed whether inference processing using the inference engine 73A can be performed. For example, if the inference engine 73A is processing another image, or if the shutter switch 62 (SW2) is still ON by operating the shutter button 60 and continuous shooting is instructed and shooting processing should take priority, the inference processing cannot be performed. If the inference processing cannot be performed, the process proceeds to S208, and if the inference processing can be performed, the process proceeds to S206.

[0039] In S206, inference processing is performed on the image file 400 using the inference engine 73A. In this embodiment, the image file 400 is input as an input to the inference engine 73A. A subject area contained in the image data is identified from the image data 406 and annotation information 403a in the image file 400, and a classification result of the subject contained in the subject area is output as a result of inference using the inference engine 73A for each subject area. During inference, in addition to the inference result, information related to the inference processing, such as debug information on the operation during the inference, logs, etc. may be output.

[0040] In S207, the subject classification result, which is the inference result obtained in S206, and the management version and debug information of the current inference model held in the inference model recording unit 72A, if any, are recorded in the image file as inference model management information. Figure 5(b) shows an image file 420 after the inference result and inference model management information have been recorded in the image file 400 in Figure 5(a). In the image file 420, the inference result is recorded in annotation information 403b for each position coordinate of the subject using the metadata generation and analysis unit 70, and inference model management information 407a is added to the MakerNote 404.

[0041] In this embodiment, an example is shown in which the annotation information is a combination of position coordinates and inference results, but the annotation information 403b may record what is output as an inference result of the inference model. For example, the output result may be recorded as it is, and the recording format or format may be any format, such as text format or binary format. By recording the image data 406 and the inference result in the same file in this way, the image data 406 and the inference result can be efficiently associated and easily managed thereafter without having to manage them separately. In addition, since the inference model management information 407a is specific data used by a manufacturer that manages the inference model, it is not widely disclosed to the public, but is recorded in a private MakerNote 404, so that it can be associated with the inference result and safely managed.

[0042] In S208, when the user instructs completion of image capture by turning off the power using the mode changeover switch or power button included in the operation unit 63, the image capture process ends.

[0043] Next, the playback process of the digital camera 100 according to this embodiment will be described with reference to FIG. In digital camera 100, when playback mode is instructed by a mode change switch included in operation unit 63, playback processing is started for viewing image data recorded on external recording medium 91. When playback processing is started, in S301, one image file to be played back is identified and read from external recording medium 91 to memory 25 via card controller 90.

[0044] In S302, the image processing unit 20 performs decompression processing, resizing processing, etc. on the image file read out to the memory 25 according to the recording method, converts it into image data for display, records it in the image display memory 24, and plays it back on the display unit 23.

[0045] In S303, it is confirmed whether the inference result has been recorded in the currently played image file using the metadata generation and analysis unit 70. If it has been recorded, the process proceeds to S304, and if it has not been recorded, the process proceeds to S309. In S304, if the user issues an instruction to display the inference results recorded in the image file using the operating members included in the operation unit 63, the process proceeds to S305, and if no instruction is issued, the process proceeds to S309.

[0046] In S305, information related to the inference result is extracted from the annotation information 403b using the metadata generation and analysis unit 70, and the extracted information is superimposed on the image data being played back using the image processing unit 20 and displayed. For example, the inference result associated with the image data is displayed so that the user can visually check it by displaying frames indicating the coordinate areas of each subject described in the annotation information 403b together with character strings indicating the inference result classifying the subjects within the coordinate areas.

[0047] In S306, it is confirmed whether the user has noticed an error as a result of referring to the inference result displayed in S305 and has issued an instruction to correct the inference result using an operating member included in the operation unit 63. If there is an instruction to correct the inference result, the process proceeds to S307, and if there is no instruction to correct the inference result, the process proceeds to S309.

[0048] In S307, an instruction to correct the inference result is obtained, and an image file 430 shown in FIG. 5(c) is created as teacher data for the inference model, and is recorded in the teacher data recording unit 74a. The image file 430 may also be recorded in the external recording medium 91 as a file separate from the image file 420. The image file 430 is created based on the image file 420, and the metadata generating and analyzing unit 70 records, as correct data, data instructed to be corrected by the user for each coordinate area within the screen of the subject in annotation information 403c recorded at a position indicated by the annotation information offset included in the annotation link information storage tag 402. The annotation information 403b of the image file 420 is recorded as old annotation information 409 at a position indicated by the old annotation information offset included in the old annotation link information storage tag 408.

[0049] Next, in S308, the learning unit 74A uses the image file 430 created in S307 as training data to re-learn the inference engine 73A, thereby updating the inference engine 73A. In conjunction with the update, the inference model recording unit 72A also updates the management version of the inference engine 73A.

[0050] In S309, it is confirmed whether the user has instructed to perform inference on the image file being played back using an operating member included in the operation unit 63. If an inference has been instructed, for example, when it is desired to perform inference processing on an image file for which inference processing could not be performed at the time of shooting, or when an inference model has been updated externally and it is desired to perform inference processing again on a recorded image file, the process proceeds to S310.

[0051] In S310, it is confirmed whether the inference result has already been recorded in the image file for which inference was instructed, and if not, in S312 and S313, the same processes as in S206 and S207 are performed to record the inference result and inference model management information in the image file. If the inference result has already been recorded, the process proceeds to S311 and re-inference processing is performed.

[0052] Here, the re-inference process performed in S311 will be described with reference to FIG. In the re-inference process, in S401, the metadata generation and analysis unit 70 is used to extract inference model management information recorded in the image file for which inference was instructed. Then, the management version included in the inference model management information is compared with the old or new management version of the inference engine 73A managed by the inference model recording unit 72A to confirm whether the management version of the inference engine 73A has been updated. If the management version of the inference engine 73A is the same or older than the management version recorded in the image file, no processing is performed and the process ends. If it has been updated, proceed to S402.

[0053] In S402, the same process as in S206 is performed. For example, when performing re-inference processing on the image file 420, the subject region included in the image data 406 is identified from the image data 406 and annotation information 403b in the image file 420. Then, inference is performed for each subject region using the inference engine 73A, and inference results are obtained for each subject region in S403.

[0054] In S404, it is determined whether the inference result acquired in S403 has a different output format from the inference result of the annotation information 403b recorded in the image file 420. A different output format may occur, for example, when the inference engine 73A is updated, making more detailed classification possible, and new items such as subitems are added to the classification results and output. If the output format is different, the process proceeds to S406.

[0055] On the other hand, if the output format is the same, in S405, the management version of the inference engine 73A managed by the inference model recording unit 72A is checked to see if it indicates that the inference model has been updated from outside. If there has been an update from outside, proceed to S406.

[0056] In S406, the metadata generating / analyzing unit 70 is used to convert the image file 440 into a format like that shown in Fig. 5(d) and record it on the external recording medium 91. In the image file 440, the result of the re-inference is recorded in annotation information 403d recorded at a position indicated by the annotation information offset included in the annotation link information storage tag 402 for the original image file 420.

[0057] On the other hand, annotation information 403b that was already recorded in the image file 420 is recorded as old annotation information 409 at the position indicated by the old annotation information offset included in the old annotation link information storage tag 408. This makes it possible to record both the old and new inference results in association with the image data, and makes it easy to manage the transition of the inference results for the image data.

[0058] In addition, in this embodiment, an inference model for inferring the classification of a subject is assumed, but there are various inference models other than classification models. Even when performing inference processing using different inference models, by using this embodiment, it is possible to efficiently associate inference results of different inference models for one image data within one file and easily manage them.

[0059] If it is determined in S405 that the inference model has not been updated from outside, in S407, the metadata generation and analysis unit 70 is used to update only the inference result portion of the annotation information 403b of the image file 420, and the process proceeds to S408.

[0060] In S408, the metadata generation and analysis unit 70 updates the inference model management information 407a in the MakerNote 404 of the image file 440 to the inference model management information 407b, which is the information of the current inference engine 73A.

[0061] 3, if an instruction to play another image is given in S314 using an operating member included in the operation unit 63, the process returns to S301 and repeats the above-mentioned process for the specified image to be played. On the other hand, if an instruction to play another image is not given, the process determines in S315 whether an instruction to end the playback process is given, and if an instruction to end the playback process is not given, the process returns to S303 and repeats the above-mentioned process, and if an instruction to end the playback process is given, the playback process ends.

[0062] Next, the transmission process of the digital camera 100 in this embodiment will be described with reference to FIG. When an instruction to transmit is given using an operating member included in the operating unit 63, the transmission process shown in the flowchart of Fig. 6 is started. The instruction to transmit may be given by selecting one or a plurality of image files as targets and starting transmission, or may be set in advance to automatically transmit when an image is captured, and transmission may be started when an image file is generated as a trigger. Here, an example will be described in which the image file before transmission is recorded in a format like the image file 440 in Fig. 5(d) described above.

[0063] In S601, a request for system information is made to the external device 101 via the communication unit 71A of the digital camera 100, and the external device 101 transmits the system information from the system control unit 50B via the communication unit 71B to the digital camera 100. As a result, the digital camera 100 obtains the system information of the transmission destination and proceeds to S602.

[0064] Next, in S602, the digital camera 100 requests inference model management information from the external device 101 via the communication unit 71A, and the external device 101 transmits the inference model management information from the system control unit 50B via the communication unit 71B to the digital camera 100. As a result, the digital camera 100 obtains the inference model management information of the destination and proceeds to S603.

[0065] In S603, a transmission method is determined. Specific examples of the transmission method include wireless transmission such as Wi-Fi or BlueTooth (registered trademark), wired transmission using an Ethernet cable or USB cable, and transmission using a removable medium such as an SD card. When a plurality of transmission methods are available, the transmission method may be determined by a method set by the user using an operation member included in the operation unit 63, or may be determined from the connection state between the external device 101 and the digital camera 100. When only a single transmission method is available, that transmission method may be determined. Once the transmission method is determined, the process proceeds to S604.

[0066] In S604, the destination system information acquired in S601 is compared with the own system information, and if they are the same, the process proceeds to S605. If they are different or if the destination system information could not be acquired, the process proceeds to S608.

[0067] In S605, the management version of the destination inference model acquired in S602 is compared with the management version of the inference model of the digital camera 100 itself held in the inference model recording unit 72A. If the destination management version and its own management version match, the process proceeds to S606, and if the destination management version and its own management version do not match or if it is not possible to determine this for some reason, such as the destination management version being unable to be acquired, the process proceeds to S607.

[0068] In S606, annotation information other than the matching management version is deleted. For example, if a match occurs with a management version other than the latest, the annotation link information storage tag 402 and annotation information 403d are deleted from the image file 440 shown in Fig. 5(d) to generate an image file 700 as shown in Fig. 7(a). By deleting annotation information other than the matching management version, annotation information that is unlikely to be used is deleted while annotation information that is used in the destination system is retained, which is expected to reduce the amount of data, improve data efficiency, and ensure scalability. Then, the process proceeds to S610.

[0069] On the other hand, if the management versions do not match, in S607, annotation information other than the latest management version is deleted. For example, the old annotation link information storage tag 408 and the old annotation information 409 are deleted from the image file 440 shown in Fig. 5(d) to generate an image file 710 as shown in Fig. 7(b). By deleting annotation information other than the latest management version, annotation information that is unlikely to be used can be deleted while retaining annotation information to be used in the destination system, thereby reducing the amount of data, improving data efficiency, and ensuring scalability. Then, the process proceeds to S610.

[0070] Furthermore, if the system information does not match, in S608, annotation information other than the latest management version is deleted, similar to the process in S607. For example, an image file 710 as shown in Fig. 7(b) is generated from the image file 440 shown in Fig. 5(d). By deleting annotation information other than the latest management version, annotation information that is unlikely to be used can be deleted, reducing the amount of data, improving data efficiency, and ensuring scalability. Then, proceed to S609.

[0071] In S609, the latest inference result is deleted. For example, the inference result included in the annotation information 705 is deleted from the image file 710 shown in Fig. 7(b), and an image file 720 including annotation information 705b and MakerNote 706 as shown in Fig. 7(c) is generated. By deleting the latest inference result, it is possible to delete inference results that are unlikely to be used, thereby reducing the amount of data, improving data efficiency, and ensuring scalability. Then, the process proceeds to S610.

[0072] In S610, it is determined whether the reliability of the transmission method determined in S603 is sufficiently high, and if it is determined to be sufficiently high, the process proceeds to S612. If it is determined that the transmission method is not sufficiently reliable, the process proceeds to S611. Specifically, if the transmission method is wired or using an SD card, it may be determined to be highly reliable, and if it is wireless, it may be determined to be unreliable. Alternatively, if it is wireless, it may be determined to be highly reliable if it is an in-house LAN, and if it is public wireless, it may be determined to be unreliable.

[0073] In S611, the annotation link information storage tag is deleted. For example, if the image file 700 shown in Fig. 7(a) was generated in S606, the old annotation link information storage tag 408 is deleted, and an image file 730 as shown in Fig. 7(d) is generated. If the image file 710 shown in Fig. 7(b) was generated in S607, the annotation link information storage tag 402 is deleted, and an image file 740 as shown in Fig. 7(e) is generated. If the image file 720 shown in Fig. 7(c) was generated in S609, the annotation link information storage tag 402 is deleted, and an image file 740 as shown in Fig. 7(f) is generated.

[0074] In this way, by deleting the annotation link information storage tag, even if the image file is intercepted during transmission, the inference results cannot be easily accessed, so it is possible to prevent the leakage of know-how and assets (data created at great expense).Then, the process proceeds to S612.

[0075] In S612, the transmission process of the target image file is performed using the transmission method determined in S603, and the process ends.

[0076] In the above explanation, the inference results are deleted for each image file and then the transmission process is performed, but it is also possible to perform the deletion process of the inference results for multiple image files together and then transmit the deleted image files together.

[0077] Next, the editing process of the digital camera 100 in this embodiment will be described with reference to FIG. When editing is instructed using the operating members included in the operation unit 63, the editing process starts as shown in the flowchart of Fig. 8. Editing can be instructed by selecting one or a plurality of image files as targets and instructing the editing contents, or by instructing the editing contents for an image displayed in a quick review on the display unit at the time of shooting.

[0078] In S801, the system control unit 50A acquires image data, Exif data 401, etc. from an image file stored in the external recording medium 91, issues a display request via the display unit 23, and proceeds to S802. In S802, the acquired image data is edited according to the editing content instructed, and the edited image data is saved. For example, if the image file before editing is recorded in a format like the image file 440 in FIG. 5(d) described above, editing is performed on the acquired image data 406, and the edited image data 912 is saved. Also, if the MakerNote 404 of the image data 406 contains information 405 about the detected subject, the subject information is converted and recorded according to the editing content. For example, if the image size has changed due to editing, the coordinates of each subject in the MakerNote 404 are converted to match the edited image size, and the converted coordinate information 911 is recorded, and the image file 910 shown in FIG. 9(a) is generated.

[0079] Next, in S803, the system control unit 50A determines whether or not annotation information is stored in the image file of the image data to be edited acquired in S801. If annotation information is stored, the process proceeds to S804, and if annotation information is not stored, the process proceeds to S820, where the edited image file is recorded and the editing process ends.

[0080] In S804, an inference process using the inference engine 73A is performed on the image data edited in S802. For example, when an image file 910 shown in FIG. 9(a) is input as an input to the inference engine 73A, first, a subject area included in the image data 912 is identified from the image data 912 in the image file 910 and the annotation information 403d. Then, as a result of inference using the inference engine 73A for each subject area, a classification result of the subject included in the subject area is output. It is also possible to use the inference engine 73B of the external device 101. Also, during inference, in addition to the inference result, information related to the inference process, such as debug information on the operation during inference and a log, may be output. When the inference process is completed, the process proceeds to S805.

[0081] In S805, if there is a management version or debug information of the current inference model held in the inference model recording unit 72A, the latest information is obtained from the inference model management information. The obtained information is then recorded in MakerNote, and the inference result of S804 is recorded as annotation information, and the existing annotation information is recorded as old annotation information. For example, for the image file 910 shown in FIG. 9(a), the inference result of S804 is recorded in annotation information 403e recorded at the position indicated by the annotation information offset included in the annotation link information storage tag 402. Meanwhile, annotation information 403b already recorded in the image file 910 is recorded as old annotation information A 409a at the position indicated by the old annotation information offset included in the old annotation link information A storage tag 408. Also, the old annotation information 409 is recorded in the old annotation link information B storage tag 90. to 8 The old annotation information B 409b is recorded at the location indicated by the included old annotation information offset. Furthermore, the management version and debug information of the current inference model are recorded as inference model management information 927 of MakerNote 404. This generates an image file 920 shown in FIG. 9(b).

[0082] Next, in S806, it is determined whether the editing process was a resizing process for enlarging or reducing the size of image data, screen display elements, etc. If it was a resizing process, the process proceeds to S814, and if it was not a resizing process, the process proceeds to S807.

[0083] In S814, the metadata generating / analyzing unit 70 is used to delete the annotation information obtained in the inference process in S804 from the image data of the image file generated in S805, and the other annotation information is retained. This is because in the case of resizing, the pixels after editing become coarse due to size conversion, so that the inference result of the original image is more accurate than inference from the edited image. For example, in the case of the image file 920 shown in FIG. 9(b), the annotation link information storage tag 402 and annotation information 403e are deleted. Meanwhile, the old annotation link information A storage tag 408, the old annotation link information B storage tag 908, the old annotation information A 409a, and the old annotation information B 409b are retained. As a result, an image file 930 as shown in FIG. 9(c) is generated. Then, the process proceeds to S817.

[0084] In S807, it is determined whether the editing process was a trimming process for adjusting the display range or size of the image by cutting off unnecessary parts around the image data. If it was a trimming process, the process proceeds to S808, and if it was not a trimming process, the process proceeds to S811.

[0085] In S808, for the subject indicated in the newer old annotation information, it is determined from the coordinate information whether or not all of the subject areas are within the area cut by the trimming process. If all of the subject areas are within the area cut, the process proceeds to S809, and if not, the process proceeds to S810.

[0086] In S809, for the subject indicated in the older one of the old annotation information, it is determined from the coordinate information whether or not all of the subject areas are within the area cut by the trimming process. If all of the subject areas are within the area cut, the process proceeds to S815, and if not, the process proceeds to S816.

[0087] In S815, the metadata generating / analyzing unit 70 is used to delete all old annotation information from the image data of the image file generated in S805, and the latest annotation information is retained. This is because the subject areas indicated in all old annotation information have been cut off by cropping. For example, in the case of the image file 920 shown in FIG. 9(b), the old annotation link information A storage tag 408, the old annotation link information B storage tag 908, the old annotation information A 409a, and the old annotation information B 409b are deleted. As a result, an image file 940 as shown in FIG. 9(d) is generated. Then, the process proceeds to S817.

[0088] In S816, the newer annotation information is deleted from the image data of the image file generated in S805 using the metadata generating / analyzing unit 70, and the latest annotation information and the older annotation information are retained. This is because the area of ​​the subject indicated in the newer annotation information has been cut off by cropping. For example, in the case of the image file 920 shown in FIG. 9(b), the old annotation link information A storage tag 408 and the old annotation information A 409a are deleted. As a result, an image file 950 as shown in FIG. 9(e) is generated. Then, the process proceeds to S817.

[0089] In S810, for the subject indicated in the old annotation information of the older image file, it is determined from the coordinate information whether or not all of the subject areas are within the area cut by the trimming process. If all of the subject areas are within the area cut, the process proceeds to S817, and if not, the process proceeds to S818.

[0090] In S817, the metadata generating / analyzing unit 70 is used to delete the older annotation information from the image data of the image file generated in S805, and the latest annotation information and the newer annotation information are retained. This is because the area of ​​the subject indicated in the older annotation information was cut off by cropping. For example, in the case of the image file 920 shown in FIG. 9(b), the old annotation link information B storage tag 908 and the old annotation information B 409b are deleted. As a result, an image file 960 as shown in FIG. 9(f) is generated. Then, the process proceeds to S817.

[0091] In S811, it is determined whether the editing process was a moving image cut-out process in which a specified frame is cut out from a moving image format such as MPEG-4, converted into JPEG format, and saved. If it was a moving image cut-out process, the process proceeds to S818, and if it was not a moving image cut-out process, the process proceeds to S812.

[0092] In S812, it is determined whether the editing process was a RAW development process. In the RAW development process, the image processing unit 20 performs compression processing such as lossless compression on uncompressed RAW data, or performs image processing such as color conversion on image data that has been decompressed, and converts the data into JPEG format to create image data. If it is a RAW development process, the process proceeds to S813, and if it is not a RAW development process, the process proceeds to S818.

[0093] In S813, it is determined whether the color of the image data has changed when the RAW development process was performed in S812. If the color of the image data has changed, the process proceeds to S814, where the above-mentioned process is performed. On the other hand, if the color of the image data has not changed, the process proceeds to S818.

[0094] In S818, all image data of the image file generated in S805 is held. For example, in the case of the image file 920 shown in Fig. 9B, the image file 920 is left as it is without being changed. Then, the process proceeds to S817.

[0095] In S817, the image files generated in S814 to S818 are recorded in the external recording medium 91, and the editing process ends.

[0096] In the above-described editing process, annotation information is deleted after inference processing is performed. However, in the case of editing content that deletes the latest annotation information (resizing processing in the example of Figure 8), inference processing may not be performed.

[0097] In the above example, resizing, trimming, video clipping, and ROW development are described as examples of editing processes, but other processes may be used. In that case, it is sufficient to delete unnecessary annotation information while leaving necessary annotation information, as in S814 to S818.

[0098] 9 shows a case where there are two pieces of old annotation information, but the present invention is not limited to this. In this case, the processes of S808 to S810 may be changed according to the number of pieces of annotation information.

[0099] Furthermore, instead of storing the data in a file format in the memory 25, each data may be stored individually, and when recorded in S817, the data may become image files such as those shown in FIGS. 9(c) to (f).

[0100] In addition, although the above description has been given of performing editing processing on an image file and then performing inference processing, the processing order may be changed so that editing processing is performed on multiple image files together and then inference processing is performed on multiple image files together.

[0101] As described above, by deleting unnecessary annotation information while leaving necessary annotation information according to the editing content, it is possible to manage annotation information while reducing the amount of data.

[0102] <Other embodiments> The present invention may be applied to a system consisting of multiple devices (such as a host computer, interface device, scanner, video camera, etc.), or to an apparatus consisting of a single device (such as a copier, facsimile machine, etc.).

[0103] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0104] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0105] 13: imaging element, 50A, 50B: system control unit, 70: metadata generation and analysis unit, 71A, 71B: communication unit, 72A, 72B: inference model recording unit, 73A, 73B: inference engine, 73a: neural network design, 74A, 74B: learning unit, 74a: teacher data recording unit

Claims

1. A detection means for detecting a subject from an image; an inference means for performing an inference process on the detected subject by using an inference model; a generating means for generating an image file by combining image data of the image, information on the subject, an inference result of the inference process, and information on the inference model; An image processing device characterized in that the generation means records the inference model in a private area of ​​the image file.

2. an operating means for correcting the inference result; A learning means for updating the inference model; Further comprising: The image processing device described in claim 1, characterized in that when the inference result is corrected by the operation means, the generation means changes the inference result of the image file to the corrected content, and the learning means updates the inference model using the corrected inference result.

3. The method further includes a control means for causing a display means to display the inference result in association with the corresponding subject, 3. The image processing apparatus according to claim 2, wherein said operation means corrects said displayed inference result.

4. The image processing device described in any one of claims 1 to 3, characterized in that the inference means is further capable of performing the inference processing on an image stored in an image file, and when information of the inference model is stored in the image file, determines whether the inference model of the image file is newer than the inference model of the inference means, and performs the inference processing if the inference model of the inference means is newer.

5. 5. The image processing apparatus according to claim 4, wherein when an output format of the inference result of the image file differs from an output format of the inference result of the inference means, the inference result of the inference means is added to the image file.

6. The inference model is externally updatable, An image processing device as described in claim 4 or 5, characterized in that when the output format of the inference result of the image file is the same as the output format of the inference result by the inference means and the inference model has been updated from outside, the inference result by the inference means is added to the image file.

7. An image processing device as described in claim 6, characterized in that when the output format of the inference result of the image file is the same as the output format of the inference result by the inference means and the inference model has not been updated from outside, the inference result recorded in the image file is updated with the inference result by the inference means.

8. A communication means for transmitting the image file; An acquisition means for acquiring information regarding an inference means to which the image file is to be transmitted; a deletion means for deleting at least a part of the inference results included in the image file in accordance with the inference means of the destination; 8. The image processing apparatus according to claim 1, wherein the communication means transmits the image file from which at least a part of the inference result has been deleted by the deletion means.

9. 9. The image processing apparatus according to claim 8, wherein said deletion means deletes an inference result in which said inference means and the inference means of the transmission destination do not match.

10. The generating means further records link information to the inference result in the image file, 10. The image processing apparatus according to claim 8, wherein the deletion means deletes the link information when the reliability of the communication means is lower than a predetermined reliability.

11. 11. The image processing apparatus according to claim 10, wherein the deletion means deletes the link information when the image file is transmitted by wireless communication.

12. The image processing device further includes an image processing unit for performing an editing process on the image stored in the image file, The inference means performs the inference process on the edited image, The image processing device according to any one of claims 1 to 11, characterized in that the generation means does not retain at least a portion of the inference results of the edited image and the inference results contained in the image file in the image file after the edit process, depending on the content of the edit process.

13. An image processing device as described in Claim 12, characterized in that in the case of content of the editing process not retaining the inference result of the edited image as an image file after the editing process, the inference means does not perform the inference process on the edited image.

14. 14. The image processing device according to claim 12, wherein the generating means, when the editing process is a resizing process, does not hold the inference result of the edited image in the image file after the editing process.

15. 15. The image processing device according to claim 12, wherein, when the editing process is a cropping process, the generating means does not retain an inference result of a subject in an area of ​​an image that is deleted by the cropping process in an image file after the editing process.

16. The image processing device according to any one of claims 12 to 15, characterized in that, when the editing process is a video clipping process, the generation means holds the inference result of the edited image and the inference result contained in the image file in the image file after the editing process.

17. The image processing device according to any one of claims 12 to 16, characterized in that when the editing process is a RAW development process and the color tone of the image does not change as a result of the development, the generation means retains the inference result of the edited image and the inference result contained in the image file in the image file after the editing process, and when the color tone changes, does not retain the inference result of the image contained in the image file in the image file after the editing process.

18. An imaging means for capturing the image; An image processing device according to any one of claims 1 to 17, An imaging device comprising:

19. A detection step in which a detection means detects a subject from an image; an inference step in which an inference means performs an inference process on the detected subject by using an inference model; A generation step in which a generation means compiles image data of the image, information on the subject, an inference result of the inference process, and information on the inference model to generate an image file; An image processing method characterized in that, in the generation process, the inference model is recorded in a private area of ​​the image file.

20. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 17.

21. A computer-readable storage medium storing the program according to claim 20.

Images