Image processing device, control method thereof, and program

Abstract

To provide an image processing device capable of improving object detection accuracy while using a learning model with the same structure.SOLUTION: An image processing device comprises a learning unit for learning a learning model using first training data including a first region labeled with a first classification for an input image, an inference unit for inferring using the learning model trained and verification data, a generating unit, and a control unit for making the learning unit relearn using second training data, where the generating unit labels the first region with the second classification subdivided from the first classification label and generates second training data including the first region labeled with the second classification when the accuracy of the inferred results by the inference unit is less than a first threshold.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device that detects a specific object from an image. [Background technology]

[0002] In recent years, many methods have been proposed for detecting specific objects from images using machine learning. To create a trained model, training data must be created that annotates training images with information on the position and label of the object to be detected, and parameters must be trained using a training program. When object detection is performed using this trained model, it may output an incorrect label for a certain object. In particular, if there is a large variation in the features of objects with the same label in the training images, parameter training may not work well, resulting in reduced inference accuracy.

[0003] For example, if you want to create a trained model to detect multiple types of lesions from images in a medical setting, creating training data using the names of the lesions as labels will result in assigning the same label to lesions that look very different depending on their stage of progression, location of appearance, etc. This can result in low detection accuracy.

[0004] Patent Document 1 proposes a technology for improving detection accuracy in a hierarchical neural network. The overall accuracy is improved by extracting misclassified data from a trained model that has already been generated, adding a layer for determining and classifying data that is easily misclassified, and retraining the model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-51589 Summary of the Invention [Problem to be solved by the invention]

[0006] The method disclosed in Patent Document 1 has a problem in that changing the structure of a trained model may increase the data size of the model and the amount of calculation required for inference.

[0007] Furthermore, when creating training data, it may be possible to improve accuracy by assigning different labels to images with different visual characteristics, but this requires workers to visually inspect the training images, classify them by visual characteristics, and re-assign the labels, which is a significant amount of work.

[0008] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can improve the accuracy of object detection while using a learning model with the same structure. [Means for solving the problem]

[0009] The image processing device according to the present invention performs a learning model using first training data including a first region in an input image labeled with a first classification. With teacher a learning means for performing learning; an inference means for performing inference using the learned learning model and verification data; a generation means for labeling the first region with a label of a second classification obtained by subdividing the label of the first classification when the accuracy of the inference result by the inference means is equal to or lower than a first threshold, and generating second teacher data including the first region labeled with the label of the second classification; Supervised learning Re degree and a control means for causing the [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the accuracy of object detection while using a learning model with the same structure. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a system configuration diagram of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating labels of detection target objects in the first embodiment. [Figure 3] FIG. 2 is a diagram showing the structures of training data and verification data according to the first embodiment. [Figure 4] 1 is a flowchart showing a process for generating a trained model according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of the screen configuration of a user interface according to the second embodiment. [Figure 6] 10 is a flowchart showing a process for generating a trained model according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] (First embodiment) In this embodiment, an image processing device is described that generates a trained model for detecting the position and type of a plurality of lesions that have been set as detection targets from an image. As an inference method, in this embodiment, a machine learning algorithm such as deep learning is used. Note that, although the detection target in this embodiment is a lesion, the object that is the detection target of the present invention is not limited to this.

[0014] FIG. 1 is a system configuration diagram of an image processing apparatus 100 according to a first embodiment of the present invention.

[0015] In FIG. 1, a central processing unit (hereinafter referred to as CPU) 101 executes programs to control the entire image processing device 100. A read only memory (hereinafter referred to as ROM) 102 stores programs and parameters. In this embodiment, the ROM 102 stores program code of software executed by the CPU 101, necessary parameters, etc. The CPU 101 executes this program code. Note that the ROM 102 in this embodiment is a flash ROM, and the control program is rewritable.

[0016] Random Access Memory (hereinafter referred to as RAM) 103 temporarily stores programs and data supplied from the outside. It is also used as a temporary storage area for data output as programs are executed. Display unit 104 is a display unit such as an LCD display, and displays the software's Graphical User Interface (GUI) screen, processing results, etc.

[0017] The recording medium 105 is a recording medium to / from which the image processing device 100 can read and write data. For example, the recording medium 105 is a medium capable of recording electronic data, such as an internal memory of a computer, a memory card detachably connected to a computer, a hard disk drive (HDD), a CD-ROM, an MO disk, an optical disk, or a magneto-optical disk. The recording medium 105 stores inference data, inference results, and data for generating inference data, such as training data.

[0018] The operation unit 106 is composed of a keyboard, mouse, etc., and can specify input / output data, change programs, execute or stop image processing, etc., by instructions input via the operation unit 106. The I / F (Interface) 107 is an interface for communicating with external systems. The internal bus 108 is a transmission path for control signals and data signals between each element.

[0019] Each function of the image processing device 100 is realized by the CPU 101 performing calculations by loading a predetermined program onto hardware such as the CPU 101 and ROM 102. Each function is also realized by controlling communication via the I / F 107 and reading and writing of data from and to the RAM 103 and recording medium 105.

[0020] In this embodiment, for ease of understanding, an example in which a CPU is installed as the main control unit of the image processing device will be described; however, the present invention is not limited to this. For example, in addition to the CPU, a graphics processing unit (GPU) may also be installed, and the CPU and GPU may work together to perform processing. Since a GPU can perform efficient calculations by processing a larger amount of data in parallel, it is effective to use a GPU for processing when performing learning multiple times using a learning model such as deep learning. Specifically, when executing a learning program including a learning model, learning is performed by the CPU and GPU working together to perform calculations. Note that the calculation processing of the learning unit may be performed only by the CPU or the GPU. Furthermore, the processing of the estimation unit may also be performed using a GPU, similar to the processing of the learning unit.

[0021] 2A and 2B are diagrams for explaining labels of detection target objects in an input image. FIG. 2A shows a label list 200. The label list 200 is composed of a combination of a label and a lesion name, and indicates that the label "AAA" is a label representing lesion A. FIG. 2B shows a label list 210 updated by the trained model generation process described later. Details will be described later.

[0022] FIG. 3 shows the structure of training data and verification data. FIG. 3(A) shows an annotation information list 300 assigned to one image file. In this embodiment, the data is recorded in XML format. Image identification information 301 is information for identifying the corresponding image file, and in this embodiment, the image file name is recorded. Image size information 302 is information related to the resolution of the entire image, and in this embodiment, the number of pixels in both the vertical and horizontal directions of the entire image is recorded.

[0023] The annotation information 303 is annotation information for the detection target object, and is composed of position information and a label within the image. In this embodiment, the position information records the coordinate xmin of the left edge, the coordinate xmax of the right edge, the coordinate ymin of the top edge, and the coordinate ymax of the bottom edge of a rectangle surrounding the detection target object in the image. Note that the position information may be other than rectangular, for example, a circle or any other arbitrary shape, as long as it matches or can be converted into the input of the learning program and the output of the inference program. The label records one of the labels listed in the label list 200. The number of annotation information 303 recorded is the same as the number of detection target objects included in the image.

[0024] Figure 3(B) shows an image file 310. Rectangles 311 and 312 visualize the annotation information assigned to lesion A and lesion B, respectively, contained in image file 310. The actual image file 310 does not contain figures such as rectangles 311 and 312, and the annotation information is stored in a separate file as shown in Figure 3(A). The training data and verification data are composed of multiple combinations of annotation information lists 300 and image files 310.

[0025] 4 is a flowchart showing the processing of generating a trained model by the image processing device 100. The processing shown in this flowchart is realized by the CPU 101 of the image processing device 100 controlling each unit of the image processing device 100 in accordance with input signals and programs stored in the ROM 102. Unless otherwise specified, the same applies to other flowcharts showing the processing of the image processing device 100.

[0026] In step S401, the CPU 101 reads the training data having the structure described with reference to FIG.

[0027] In step S402, the CPU 101 executes a learning program using the training data read in step S401, and generates a trained model for object detection.

[0028] In step S403, the CPU 101 reads the verification data having the structure described with reference to FIG.

[0029] In step S404, the CPU 101 executes an inference program using the image file of the verification data read in step S403 as input, performs object detection, and acquires an inference result, which has the same structure as the annotation information list 300 in FIG.

[0030] In step S405, CPU 101 compares the inference result acquired in step S404 with the annotation information of the verification data read in step S403, and calculates the overall accuracy. The method of calculating the accuracy will be described later. If step S405 is being executed for the first time, or if the overall accuracy is equal to or greater than the value when step S405 was executed last time (if the accuracy has improved), CPU 101 proceeds to step S406; otherwise, CPU 101 proceeds to step S412.

[0031] In step S406, CPU 101 calculates the accuracy and the number of data included in the training data for each label listed in label list 200. Then, it is determined whether the accuracy for any label is equal to or less than a predetermined threshold set for accuracy and the number of data is equal to or greater than a predetermined threshold set for the number of data. If the accuracy for any label is equal to or less than the predetermined threshold set for accuracy and the number of data is equal to or greater than a predetermined threshold set for the number of data, CPU 101 proceeds to step S407; otherwise, it terminates the process. Each threshold may be a value predetermined by the program or a value specified by the user.

[0032] Steps S407 to S411 form a loop in which CPU 101 sequentially processes each label whose accuracy falls below the predetermined threshold set for accuracy in step S406. The following processing is performed for each label. In the following explanation, the label to be processed is assumed to be "AAA."

[0033] In step S408, the CPU 101 extracts annotation information labeled "AAA" from the list of all annotation information in the training data read in step S401, and cuts out the partial image indicated by the position information from the image file.

[0034] In step S409, CPU 101 performs clustering (subdivision) using unsupervised learning with all the partial images extracted in step S408 as input. The algorithm for unsupervised learning is not particularly limited. The number of clusters may be a value predetermined by the program or may be a value specified by the user. Alternatively, the number of clusters may also be automatically determined by the unsupervised learning algorithm. In this embodiment, the number of clusters is set to 3. As a result of this process, all the partial images are classified into three groups.

[0035] In step S410, CPU 101 updates the labels based on the clustering results of step S409. Specifically, the label names of each cluster are set to "AAA_1," "AAA_2," and "AAA_3," and the label of the annotation information in the training data from which the partial images classified into the "AAA_1" cluster were extracted is changed to "AAA_1." The label list is also updated as shown by 210 in FIG. 2. That is, a label list 201 is created that includes information that the newly created "AAA_1," "AAA_2," and "AAA_3" are all labels indicating lesion A. The inference program executed in step S404 is also changed so that the label "AAA" is output when the inference result is any of "AAA_1," "AAA_2," and "AAA_3."

[0036] In step S411, CPU 101 executes the following loop: If the process has been completed for all labels, CPU 101 returns the process to step S401.

[0037] In step S412, CPU 101 restores the labels updated when step S410 was previously executed and the trained model generated when step S402 was executed to their previous states, and ends the process.

[0038] Here, we will explain how to calculate the accuracy in steps S405 and S406 in Fig. 4. Generally, there are multiple indices for accuracy in object detection, but in this embodiment, we will use the average precision. The accuracy of the inference result is determined by comparing the coordinates of the rectangles included in the annotation information of the verification data, and if the Intersection over Union (IoU) is 0.5 or more and the labels are the same, it is considered to be correct, and if not, it is considered to be incorrect.

[0039] In step S405 of Fig. 4, CPU 101 uses the average of the average precision rates for each label, i.e., the sum of the average precision rates divided by the number of lesions, as the overall accuracy. In step S406 of Fig. 4, CPU 101 calculates the average precision rate for each lesion. Note that if the number of data included in the validation data varies depending on the lesion, the calculation may be weighted by the number of data.

[0040] As described above, according to the image processing device of this embodiment, in the process of generating a trained model for object detection, training data with labels that have low detection accuracy are subdivided using unsupervised learning, and a different label is assigned to the training data, followed by re-training. This suppresses accuracy degradation due to variations in features within the same label, and improves overall accuracy. Furthermore, by performing these processes automatically, it is possible to improve accuracy without manually updating annotation information.

[0041] (Second embodiment) In the first embodiment, an example was described in which the determination of whether to continue label updating and relearning is performed automatically. In the present embodiment, an example will be described in which a user can check the label update status and issue a relearning instruction by user operation.

[0042] In this embodiment, the description of the same parts as in the first embodiment will be omitted, and the description will mainly focus on the configuration unique to this embodiment.

[0043] FIG. 5 is a diagram showing an example of the configuration of a user interface (UI) 500 displayed on the display unit 104 by the image processing device 100. The Confirm button 501 is used to confirm the labels and trained model based on the re-learning history, and can be used to give an instruction to end training. The Continue button 502 is used by the user to give an instruction to continue training. The history list 503 displays the label list 201 for each training session, the accuracy of each lesion based on the inference results of the validation data, and part of the image of the training data to which each label has been assigned. It is assumed that the training data to be displayed is selected at random. The user can set the history to a selected state (selectable) by clicking any item in the history list 503.

[0044] FIG. 6 is a flowchart showing the process by which the image processing device 100 generates a trained model.

[0045] In steps S601 to S604, the same processes as in steps S401 to S404 in FIG. 4 are performed, respectively.

[0046] In step S620, the CPU 101 displays the UI 500 on the display unit 104. Then, the history list 503 displays the label list 201 updated in the previous step S610, part of the training data read in step S601, and the accuracy of each lesion in the inference result of step S604.

[0047] In step S605, CPU 101 accepts a user operation, and if the user presses the enter button, proceeds to step S612; otherwise, proceeds to step S606.

[0048] In step S606, CPU 101 accepts a user operation, and if the user presses the continue button, proceeds to step S607; otherwise, returns the process to step S605.

[0049] In steps S607 to S611, the same processes as in steps S407 to S411 in FIG. 4 are performed, respectively.

[0050] In step S612, based on the history that is selected in the history list 503 on the UI 500, the CPU 101 restores the label updated in step S610 and the trained model generated in step S602 to the state of the selected history, and ends the processing.

[0051] As described above, according to the image processing device of this embodiment, the user can select whether to continue relearning or return to the specified state based on their instructions, allowing the processing to be terminated at the timing desired by the user.

[0052] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0053] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0054] 100: Image processing device, 101: CPU, 102: ROM, 103: RAM, 104: Display unit, 105: Recording medium, 106: Operation unit, 107: I / F

Claims

1. a learning means for performing supervised learning of a learning model using first training data including a first region in an input image labeled with a first classification; an inference means for performing inference using the learned model and verification data; a generation means for labeling the first region with a second classification label obtained by subdividing the first classification label when the accuracy of the inference result by the inference means is equal to or less than a first threshold, and generating second training data including the first region labeled with the second classification label; control means for causing the learning means to perform supervised learning again using the second training data; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the generating means subdivides the labels of the first classification into the labels of the second classification by unsupervised learning.

3. 3. The image processing apparatus according to claim 1, wherein the control means repeats the re-learning until the accuracy of the inference result is no longer improved.

4. 4. The image processing device according to claim 1, wherein the generating means employs an average of average precision rates of labels of classification as the accuracy of the inference result.

5. 5. The image processing apparatus according to claim 1, wherein the generation means sets the number of subdivisions to a predetermined number.

6. 5. The image processing apparatus according to claim 1, wherein the generation means sets the number of subdivisions according to a user's designation.

7. 7. The image processing device according to claim 1, wherein the generating means performs the segmentation when the number of data included in the first training data is equal to or greater than a second threshold value.

8. 3. The image processing device according to claim 1, further comprising a display means for displaying the updated status of classification labels each time the learning model is trained, and a selection means for allowing the user to select whether or not to perform re-learning.

9. 9. The image processing apparatus according to claim 8, wherein the selection means allows a user to select one of the update states of the classification labels displayed on the display means.

10. A control method executed by an image processing device, comprising: a learning step of performing supervised learning of a learning model using first training data including a first region in an input image labeled with a first classification; an inference step of performing inference using the learned model and verification data; a generation step of labeling the first region with a second classification label obtained by subdividing the first classification label when the accuracy of the inference result in the inference step is equal to or less than a first threshold, and generating second training data including the first region labeled with the second classification label; a control step of causing the learning step to perform supervised learning again using the second training data; A control method comprising:

11. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 9.

Images