Machine learning device, machine learning method, machine learning program, image diagnosis device, and ultrasonic diagnosis device

Abstract

To provide a machine learning device which generates a learning model that can provide a highly reliable inference result in response to a difference in recognition region between annotators and difference in the way of thinking of a user when recognizing a specific region in a medical image by a machine learning technology.SOLUTION: A machine learning device of the present invention includes: a storage part for storing a learning model for outputting two or more inference results related to a predefined interest region with the medical image as an input; a generation part for generating two or more modified correct label images 232, 232 which are different from each other from a plurality of correct label images 221-224 which are different from each other generated on an interest region of one medical image in advance; and a learning part for having the learning model learn the one medical image and the two or more modified correct label images 231, 232 as learnt data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a machine learning device, a machine learning method, a machine learning program, an imaging diagnostic device, and an ultrasound diagnostic device. [Background technology]

[0002] Machine learning techniques, such as deep learning, are widely used in fields such as medical image analysis.

[0003] In this regard, Patent Document 1 below discloses a technique for recognizing bone surfaces and joint positions using machine learning techniques such as neural networks on ultrasound images acquired by an ultrasound diagnostic device. The technique of Patent Document 1 eliminates the need for a user to visually check ultrasound images and recognize bone surfaces and joint positions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6191328 Summary of the Invention [Problem to be solved by the invention]

[0005] When using machine learning technology to recognize specific regions on ultrasound images, it is necessary to prepare a large number of correct images in which specific regions have been labeled by experts (annotators), such as doctors, and train the learning model.

[0006] However, labeling specific regions on ultrasound images can be a challenging task depending on the target region. For example, when labeling the region corresponding to the brachial plexus on an ultrasound image of the neck, the region recognized as the brachial plexus varies slightly among experts, resulting in different shapes and sizes of labels. Furthermore, users of image recognition results generated by machine learning technology may differ in their understanding of the region corresponding to the brachial plexus, resulting in different desired image recognition results.

[0007] For this reason, when using machine learning technology to recognize specific regions in medical images, it is desirable to be able to accommodate differences in recognition regions between experts and differences in user thinking.

[0008] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a machine learning device that generates a learning model that can provide highly reliable inference results when recognizing specific regions of medical images using machine learning technology, while taking into account differences in recognition regions between annotators and differences in user thinking.

[0009] Another object of the present invention is to provide a machine learning method used in the machine learning device, and a machine learning program executed in the machine learning device.

[0010] It is still another object of the present invention to provide an image diagnostic device and an ultrasound diagnostic device equipped with a learning model generated by the above-mentioned machine learning device. [Means for solving the problem]

[0011] The above object of the present invention can be achieved by the following means.

[0012] (1) A machine learning device having: a memory unit that stores a learning model that takes a medical image as input and outputs two or more inference results regarding a specified region of interest; a generation unit that generates two or more different corrected labeled images from a plurality of different correct labeled images that have been generated in advance for the region of interest of a single medical image; and a learning unit that trains the learning model using the single medical image and the two or more corrected labeled images as training data.

[0013] (2) The machine learning device described in (1) above, wherein the learning model takes a medical image as input and outputs a first inference result and a second inference result, the generation unit generates a first corrected correct labeled image and a second corrected correct labeled image from the plurality of correct labeled images, and the learning unit trains the learning model by calculating losses for pairs of the first inference result and the first corrected correct labeled image and for pairs of the second inference result and the second corrected correct labeled image.

[0014] (3) The machine learning device described in (1) or (2) above, wherein the two or more corrected correct labeled images include a first corrected correct labeled image and a second corrected correct labeled image having a larger area of ​​the label region for the region of interest than the first corrected correct labeled image.

[0015] (4) The machine learning device according to (3) above, wherein the label region of the first corrected correct labeled image and the label region of the second corrected correct labeled image partially overlap.

[0016] (5) The machine learning device according to (1) or (2) above, wherein the plurality of correctly labeled images include images in which a label area has been set by an operator.

[0017] (6) The machine learning device according to (1) or (2) above, wherein the plurality of correctly labeled images include images in which label regions are set by another learning model.

[0018] (7) The machine learning device according to (1) or (2) above, wherein the corrected correct label image is assigned two or more values.

[0019] (8) The machine learning device according to (7) above, wherein the two or more values ​​are changed depending on an operator who sets the label area.

[0020] (9) The machine learning device described in (7) above, wherein when the learning unit trains the learning model, the weight of the loss between the corrected correct label image and the inference result is adjusted based on multiple correct label images used to generate the corrected correct label image.

[0021] (10) The machine learning device described in (1) or (2) above, wherein the learning model is a single learning model that takes a medical image as input and outputs the two or more inference results.

[0022] (11) The machine learning device according to (1) or (2) above, wherein the medical image is an ultrasound image.

[0023] (12) A machine learning method used in a machine learning device having a memory unit that stores a learning model that takes a medical image as input and outputs two or more inference results regarding a specified region of interest, the machine learning method comprising: (a) a step of generating two or more different corrected correct labeled images from a plurality of different correct labeled images that have been generated in advance for the region of interest of a single medical image; and (b) a step of training the learning model using the single medical image and the two or more corrected correct labeled images as training data.

[0024] (13) A machine learning program executed in a machine learning device having a memory unit storing a learning model that takes a medical image as input and outputs two or more inference results regarding a specified region of interest, the machine learning program causing the machine learning device to execute the following steps: (a) generating two or more different corrected labeled images from a plurality of different correct labeled images that have been generated in advance for the region of interest of a single medical image; and (b) training the learning model using the single medical image and the two or more corrected labeled images as training data.

[0025] (14) An imaging diagnostic device having a memory unit that stores a learning model that takes a medical image as input and outputs two or more inference results regarding a specified region of interest, and a display unit that displays a display image generated based on the two or more inference results, wherein the learning model is trained using two or more different corrected correct labels generated from a plurality of different correct label images previously generated for the region of interest of a single medical image, and the single medical image, as learning data.

[0026] (15) An ultrasound diagnostic device comprising: an ultrasound probe for transmitting and receiving ultrasound waves to a subject; a memory unit for storing a learning model that receives an ultrasound image acquired by the ultrasound probe as input and outputs two or more inference results regarding a predetermined region of interest; and a display unit for displaying a display image generated based on the two or more inference results, wherein the learning model is trained using two or more different corrected correct answer label images generated from a plurality of different correct answer label images previously generated for the region of interest of a single ultrasound image, and the single ultrasound image as learning data. [Effects of the Invention]

[0027] According to the present invention, when recognizing specific regions of medical images using machine learning technology, it is possible to provide highly reliable inference results by taking into account differences in recognition regions between annotators and differences in user thinking. [Brief explanation of the drawings]

[0028] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for purposes of illustration only and are not intended to define the limits of the invention. [Figure 1] 1 is a block diagram showing a schematic configuration of an ultrasound diagnostic apparatus. [Figure 2] FIG. 2 is a diagram showing the contents stored in a storage unit of the ultrasound diagnostic apparatus. [Figure 3]FIG. 1 is a block diagram showing a schematic configuration of a machine learning device. [Figure 4] FIG. 2 is a diagram illustrating the contents stored in a storage unit of the machine learning device. [Figure 5] FIG. 10 is a diagram illustrating learning data. [Figure 6] 10 is a flowchart showing the procedure of machine learning processing. [Figure 7] FIG. 10 is a diagram for explaining a corrected correct label image. [Figure 8] FIG. 10 is a diagram illustrating a learning process. [Figure 9A] FIG. 10 is a diagram for explaining a process for generating a corrected correct label image. [Figure 9B] FIG. 10 is a diagram for explaining a process for generating a corrected correct label image. [Figure 9C] FIG. 10 is a diagram for explaining a process for generating a corrected correct label image. [Figure 9D] FIG. 10 is a diagram for explaining a process for generating a corrected correct label image. [Figure 10] FIG. 10 is a diagram illustrating another example of a process for generating a corrected correct labeled image. [Figure 11] FIG. 10 is a diagram illustrating yet another example of a process for generating a corrected correct labeled image. [Figure 12A] FIG. 10 is a diagram for explaining a first modified example of the machine learning process. [Figure 12B] FIG. 10 is a diagram for explaining a first modified example of the machine learning process. [Figure 12C] FIG. 10 is a diagram for explaining a first modified example of the machine learning process. [Figure 13] FIG. 10 is a diagram for explaining a second modification of the machine learning process. DETAILED DESCRIPTION OF THE INVENTION

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings, but the scope of the present invention is not limited to the disclosed embodiments.

[0030] 1 is a block diagram showing the schematic configuration of an ultrasound diagnostic device 1 according to one embodiment of the present invention. The following description will be given taking as an example a case where an ultrasound image of the neck is acquired by the ultrasound diagnostic device 1 and the brachial plexus is recognized.

[0031] 1, the ultrasound diagnostic device 1 includes an ultrasound probe 10 and a device main body 20. The ultrasound probe 10 and the device main body 20 are connected via a cable.

[0032] The ultrasonic probe 10 transmits ultrasonic waves to the subject and receives the reflected waves. The ultrasonic probe 10 converts the received reflected waves into electrical signals and outputs them to the device main body 20.

[0033] The device main body 20 has a control unit 21, a memory unit 22, a communication unit 23, a transmission / reception unit 24, an ultrasound image generation unit 25, a display unit 26, and an operation unit 27. The control unit 21, the memory unit 22, the communication unit 23, the transmission / reception unit 24, the ultrasound image generation unit 25, the display unit 26, and the operation unit 27 are electrically connected to each other.

[0034] The control unit 21 is configured with a CPU (Central Processing Unit) and memories such as RAM (Random Access Memory) and ROM (Read Only Memory), and controls the above-mentioned units and performs various arithmetic processing according to programs.

[0035] The storage unit 22 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs and various data.

[0036] The communication unit 23 is an interface for communicating with other external devices, and uses a communication interface conforming to various wired or wireless standards.

[0037] The transmitting / receiving unit 24 transmits an electrical signal to the ultrasound probe 10, causing the ultrasound probe 10 to generate an ultrasonic wave. The transmitting / receiving unit 24 also generates an electrical signal of the reflected wave and outputs it to the ultrasound image generating unit 25.

[0038] The ultrasonic image generating unit 25 generates an ultrasonic image using the electrical signal output from the transmitting / receiving unit 24. A conventional method is used to generate an ultrasonic image (ultrasonic image data).

[0039] The display unit 26 is, for example, a liquid crystal display, and displays ultrasound images and the like on the display screen.

[0040] The operation unit 27 includes various switches, buttons, a trackball, a mouse, a keyboard, a touchpad, etc., and outputs operation signals to the control unit 21.

[0041] FIG. 2 is a diagram showing the contents stored in the storage unit 22 of the ultrasound diagnostic device 1. The storage unit 22 of the ultrasound diagnostic device 1 stores a learning model 100 that receives an ultrasound image of the neck as input and outputs two confidence images (inference results) for the brachial plexus, which is a region of interest. The learning model 100 is based on a known algorithm such as a convolutional neural network (CNN). After undergoing machine learning in the machine learning device 30, which will be described later, the learning model 100 is transferred to the ultrasound diagnostic device 1 and stored in the storage unit 22. The confidence images are images in which the confidence for the region of interest is plotted for each pixel of the ultrasound image, and show the distribution of confidence for the entire ultrasound image.

[0042] Furthermore, the storage unit 22 of the ultrasound diagnostic apparatus 1 stores programs corresponding to the post-processing unit 110 and the setting unit 120. The post-processing unit 110 calculates a weighted average of two certainty images output by the learning model 100 to generate a final image for display. The setting unit 120 sets a weight value for the two certainty images to be weighted averaged. The functions of each of the above units are realized by the control unit 21 executing the corresponding program. Furthermore, the weight value set by the setting unit 120 may be a value input by the user via the operation unit 27, or may be a value stored in advance as an examination condition.

[0043] Next, with reference to FIGS. 3 and 4, a machine learning device 30 that trains the learning model 100 through machine learning will be described.

[0044] 3 is a block diagram showing a schematic configuration of the machine learning device 30. The machine learning device 30 is a computer such as a PC (Personal Computer).

[0045] As shown in FIG. 3, the machine learning device 30 includes a control unit 31, a storage unit 32, a communication unit 33, a display unit 34, and an input unit 35, which are interconnected by a bus.

[0046] The control unit 31, storage unit 32, communication unit 33, and display unit 34 are similar to the above-mentioned units of the ultrasound diagnostic device 1, and therefore their description will be omitted. The input unit 35 includes a keyboard, numeric keypad, mouse, etc., and is used to input various types of information.

[0047] FIG. 4 is a diagram showing the contents stored in the storage unit 32 of the machine learning device 30. The storage unit 32 of the machine learning device 30 stores a learning model 100 and learning data 200. The learning model 100 is a single learning model that receives an ultrasound image of the neck as input and outputs two confidence images for the brachial plexus. The learning data 200 includes an ultrasound image 210 and first to fourth correctly labeled images 221 to 224 (see FIG. 5) in which labels are assigned to regions on the ultrasound image 210 that correspond to the brachial plexus. The first to fourth correctly labeled images 221 to 224 will be described in detail below.

[0048] Furthermore, the storage unit 32 of the machine learning device 30 stores programs corresponding to the generation unit 130 and the learning unit 140. The generation unit 130 generates first and second corrected correct labeled images 231, 232 (see FIG. 7) from the first to fourth correct labeled images 221-224. The learning unit 140 trains the learning model 100 using the ultrasound image 210 and the first and second corrected correct labeled images 231, 232 as training data. Note that the functions of the above-mentioned units are realized by the control unit 31 executing the corresponding programs.

[0049] The ultrasound diagnostic device 1 and the machine learning device 30 may include components other than those described above, or may not include some of the components described above.

[0050] The operation of the machine learning device 30 will be described below with reference to FIGS.

[0051] First, with reference to FIG. 5, the training data 200 used for machine learning will be described. An ultrasound image 210 and first to fourth correctly labeled images 221 to 224 are stored as the training data 200 in the storage unit 32 of the machine learning device 30 in a mutually associated manner. The first to fourth correctly labeled images 221 to 224 are mask images in which regions corresponding to the brachial plexus in the ultrasound image 210 have been selected as labeled regions 221a to 224a by multiple experts (annotators). Even experts have difficulty recognizing nerve regions such as the brachial plexus on an ultrasound image, and each expert has a different understanding of nerve region boundaries. Therefore, the shapes, positions, and sizes of the labeled regions 221a to 224a in the first to fourth correctly labeled images 221 to 224 do not completely match one another.

[0052] 6 is a flowchart showing the steps of the machine learning process executed by the machine learning device 30. The process shown in the flowchart in FIG. 6 is executed by the control unit 31 in accordance with a program stored in the storage unit 32 of the machine learning device 30.

[0053] (Step S101) The control unit 31 first acquires the ultrasound image 210. More specifically, the control unit 31 acquires the ultrasound image 210 stored in the storage unit 32 as training data 200.

[0054] (Step S102) Next, the control unit 31 acquires the first to fourth correctly labeled images 221 to 224 corresponding to the ultrasound image 210. More specifically, the control unit 31 acquires the first to fourth correctly labeled images 221 to 224 stored in the storage unit 32 in association with the ultrasound image 210 acquired in the processing of step S101.

[0055] (Step S103) Next, the control unit 31 generates first and second corrected correct labeled images 231, 232 from the first to fourth correct labeled images 221 to 224. More specifically, the control unit 31 generates a first corrected correct labeled image 231 having a first label region and a second corrected correct labeled image 232 having a second label region as regions corresponding to the brachial plexus from the first to fourth correct labeled images 221 to 224. The first label region and the second label region are different from each other.

[0056] FIG. 7 is a diagram illustrating the first and second corrected correct labeled images 231, 232. In this embodiment, the first and second corrected correct labeled images 231, 232 are generated from the first to fourth correct labeled images 221-224. The first corrected correct labeled image 231 has a first label region 231a as a region corresponding to the brachial plexus. The second corrected correct labeled image 232 has a second label region 232a as a region corresponding to the brachial plexus. The first label region 231a and the second label region 232a partially overlap, while the area of ​​the second label region 232a is larger than the area of ​​the first label region 231a. The process of generating the first and second corrected correct labeled images 231, 232 will be described later.

[0057] (Step S104) Then, the control unit 31 trains the learning model 100 using the ultrasound image 210 and the first and second corrected correct-label images 231, 232 as a training data set, and ends the process. More specifically, the control unit 31 trains the learning model 100 by calculating the loss between the two confidence images output by the learning model 100 in response to the input of the ultrasound image 210 and the first and second corrected correct-label images 231, 232, and ends the process.

[0058] 8 is a diagram for explaining the learning process of the learning model 100. The learning unit 140 (see FIG. 4) that trains the learning model 100 includes an evaluation unit 141 that calculates the loss between the inference result of the learning model 100 and the corrected correct label images 231, 232, and an update unit 142 that updates the parameters of the learning model 100.

[0059] As shown in FIG. 8 , learning model 100 receives one ultrasound image 210 as input and outputs first and second confidence images 241, 242 for the brachial plexus. First confidence image 241 is input to evaluation unit 141 along with first corrected correct-label image 231, and a loss (first LOSS) between first confidence image 241 and first corrected correct-label image 231 is calculated. Similarly, second confidence image 242 is input to evaluation unit 141 along with second corrected correct-label image 232, and a loss (second LOSS) between second confidence image 242 and second corrected correct-label image 232 is calculated. Then, update unit 142 updates the parameters of learning model 100, updating learning model 100, so as to minimize the weighted sum of the first LOSS and the second LOSS. The technique for training a learning model by calculating the loss is a well-known technique, and therefore a detailed description thereof will be omitted. Furthermore, the weights of the first LOSS and the second LOSS may be adjusted as appropriate.

[0060] As described above, according to the machine learning process of this embodiment, two corrected correct labeled images 231, 232 are generated from four correct labeled images 221-224 that have been generated in advance for a region recognized as the brachial plexus on one ultrasound image 210. Then, machine learning of the learning model 100 is performed using the two corrected correct labeled images 231, 232 and the ultrasound image 210 as training data.

[0061] With this configuration, a learning model 100 is generated that has been trained using two corrected correct labeled images 231, 232 with different label regions. Then, with an ultrasound diagnostic device 1 equipped with the learning model 100, the learning model 100 outputs two certainty images using an ultrasound image of the neck acquired by an ultrasound probe 10 as input. As a result, it becomes possible to generate a final display image with a high degree of freedom through post-processing based on the two certainty images.

[0062] Specifically, for example, even if the inference result of the learning model 100 differs from the recognition result desired by the user, by performing appropriate post-processing, an inference image that is close to the recognition result desired by the user can be displayed on the display unit 26. Note that, in a typical certainty image, the certainty changes sharply at the boundary of the region of interest, so it is difficult to adjust by post-processing using only one certainty image.

[0063] 9A to 9D, a process for generating the first and second corrected correct labeled images 231, 232 will be described. In this embodiment, as shown in Fig. 9A, four correct labeled images 221 to 224 having different label regions 221a to 224a are generated in advance for one ultrasound image 210. Then, two corrected correct labeled images 231, 232 are generated from the four correct labeled images 221 to 224 generated in advance.

[0064] Specifically, first, three correct labeled images are sequentially selected from the four correct labeled images 221-224, and four groups are generated. As shown in Fig. 9B, the first group includes the first to third correct labeled images 221-223 (label regions 221a-223a), the second group includes the first, second, and fourth correct labeled images 221, 222, and 224 (label regions 221a, 222a, and 224a). The third group includes the second to fourth correct labeled images 222-224 (label regions 222a-224a), and the fourth group includes the first, third, and fourth correct labeled images 221, 223, and 224 (label regions 221a, 223a, and 224a).

[0065] Then, for each of the first to fourth groups, the label area common to the label areas of the three correctly labeled images is calculated as an AND area. As a result, as shown in FIG. 9C, first to fourth AND areas 311 to 314 are calculated for the first to fourth groups. The first AND area 311 is the area common to the label areas 221a to 223a of the first to third correctly labeled images 221 to 223. The second AND area 312 is the area common to the label areas 221a, 222a, and 224a of the first, second, and fourth correctly labeled images 221, 222, and 224. The third AND area 313 is the area common to the label areas 222a to 224a of the second to fourth correctly labeled images 222 to 224. The fourth AND region 314 is a region common to the label regions 221a, 223a, and 224a of the first, third, and fourth correct labeled images 221, 223, and 224.

[0066] 9D, the second AND region 312, which has the largest area, is selected from the first to fourth AND regions 311 to 314, and a first corrected correct labeled image 231 is generated with the second AND region 312 as the label region 231a. The first corrected correct labeled image 231 generated in this manner has a label region 231a with higher certainty, thereby reducing false positives in the learning model 100.

[0067] 9D, a region including all of the first to fourth AND regions 311 to 314 (the sum of the first to fourth AND regions 311 to 314) is calculated. Then, a second corrected correct labeled image 232 is generated, in which the region including all of the first to fourth AND regions 311 to 314 is defined as the label region 232a. The second corrected correct labeled image 232 generated in this manner has a relatively wide label region 232a, and therefore the detection performance of the learning model 100 can be improved.

[0068] The learning model 100, which has been trained using the first and second corrected correct-label images 231, 232 as training data, takes an ultrasound image of the neck as input and outputs first and second certainty images. The first certainty image has a more certain label area, similar to the first corrected correct-label image 231, and the second certainty image has a relatively wide label area, similar to the second corrected correct-label image 232. The user of the ultrasound diagnostic device 1 can set weights for the first and second certainty images according to their own way of thinking and confirm the final estimation result (display image) obtained by taking the weighted average of the first certainty image and the second certainty image.

[0069] In the above-described embodiment, an area including all four AND areas 311 to 314 (the sum of the four AND areas 311 to 314) is calculated as the label area 232a of the second corrected correct labeled image 232. However, the label area 232a of the second corrected correct labeled image 232 may be, for example, an area including two or three AND areas selected from the four AND areas 311 to 314.

[0070] Specifically, for example, as shown in FIG. 10 , from among the four AND regions 311-314, an AND region 313 is excluded because the matching rate of the label regions of the three correctly labeled images is less than a predetermined threshold. Then, an region including all three remaining AND regions 311, 312, and 314, whose matching rates of the label regions of the three correctly labeled images are equal to or greater than the predetermined threshold, is set as the label region 232a of the second corrected correctly labeled image 232. Any index may be used for the matching rate of the label regions. This configuration can reduce the influence of mislabels and generate a more reliable corrected correctly labeled image, thereby improving the diagnostic accuracy of the learning model 100. Note that even highly skilled annotators may occasionally select the wrong label region, resulting in mislabels.

[0071] Furthermore, the matching rate of the label regions of two of the three correctly labeled images constituting each group may be used as an index for selecting an AND region to be used as the label region 232a of the second corrected correctly labeled image 232. Specifically, as shown in Fig. 11, two label regions are sequentially selected from the label regions 221a to 223a of the three correctly labeled images. Then, if the matching rates (e.g., Dice coefficients) of the two selected label regions are all equal to or greater than a predetermined threshold, the corresponding AND region is used as the label region 232a of the second corrected correctly labeled image 232.

[0072] In the above-described embodiment, the AND regions 311 to 314 are calculated for each of the three correct labeled images 221 to 224, and the first and second corrected correct labeled images 231 and 232 are generated based on the four AND regions 311 to 314. However, unlike the above-described embodiment, the first and second corrected correct labeled images 231 and 232 may be generated independently of the four AND regions 311 to 314. For example, the label region 231a of the first corrected correct labeled image 231 may be an AND region common to the label regions 221a to 224a of the four correct labeled images 221 to 224. Similarly, the label region 232a of the second corrected correct labeled image 232 may be an OR region (the sum of the label regions 221a to 224a) that includes all of the label regions 221a to 224a of the four correct labeled images 221 to 224.

[0073] In this embodiment, the value "0" is set to the areas other than the label areas 221a-224a of the correct labeled images 221-224. On the other hand, only one value such as "1" may be set to the label areas 221a-224a of the correct labeled images 221-224, or two or more values ​​or consecutive values ​​may be set. For example, when a value such as "1" is assigned to the label areas 221a-224a of the correct labeled images 221-224, a value such as "1" is usually also assigned to the label areas 231a, 232a of the corrected correct labeled images 231, 232. In other words, when two values, "1" and "0", are assigned to the correct labeled images 221-224, two values, "1" and "0", are usually also assigned to the corrected correct labeled images 231, 232.

[0074] (Variation 1) Next, a first modification of the machine learning process will be described with reference to Figures 12A to 12C. In this modification, three or more values ​​are assigned to the corrected correct labeled image.

[0075] Specifically, for example, when generating a second corrected correct labeled image 232 from three correct labeled images 221-223 in which three annotators have selected label regions 221a-223a, different values ​​are set according to the degree of overlap of the label regions 221a-223a. More specifically, as shown in FIG. 12A, a value of "1" is set for regions selected by all three annotators, and a value of "2 / 3" or a value of "0.5" is set for regions selected by two annotators. A value of "1 / 3" or a value of "0.1" is set for regions selected by only one annotator. A value of "0" is set for the remaining regions other than the label regions. With this configuration, a second corrected correct labeled image 232 with higher reliability can be generated.

[0076] The above three values ​​may be changed depending on the proficiency of the annotator who selects the labeled region. Specifically, as shown in FIG. 12B, for example, for a region corresponding to the labeled region 222a selected by a highly skilled annotator, the values ​​"0.5" and "0.1" are changed to "1.0" and "0.7," respectively. In addition, for a region selected by only one of the remaining two annotators, the value "0.1" is changed to "0.3." This configuration can suppress the influence of less skilled annotators.

[0077] In addition, spatial smoothing may be performed on the three values ​​above to assign more precise values ​​to the correct labeled image. Specific coordinates (e.g., centroids) calculated based on the distribution of values ​​may be used as new labeled regions, and this method may be applied to detection tasks other than segmentation.

[0078] Furthermore, three or more values ​​may be assigned to the first corrected correct labeled image 231. Specifically, as shown in Fig. 12C, for example, a value of "1" is set for a region selected by all three annotators, a value of "1 / 3" is set for a region selected by two annotators, and a value of "0" is set for a region selected by only one annotator.

[0079] Unlike this modification, when the second corrected correct labeled image 232 is generated from three correct labeled images 221-223 in which three annotators have selected label regions 221a-223a, two values ​​may be assigned to the second corrected correct labeled image 232. In this case, for example, the value "1" is assigned to regions selected by all three annotators and regions selected by two annotators, and the value "0" is assigned to regions selected by only one annotator and the remaining regions other than the label regions. With this configuration, it is possible to generate a second corrected correct labeled image 232 with higher reliability (which is expected to reduce the influence of mislabels, and in particular reduce false positives).

[0080] Furthermore, even when only two values ​​are assigned to a corrected correct labeled image, the value assigned to the corrected correct labeled image may be changed depending on the proficiency of the annotator. Specifically, for example, for a corrected correct labeled image generated from a plurality of correct labeled images in which a plurality of highly skilled annotators have selected label regions, the value "1" is set to the label region and the value "0" is set to the remaining regions. On the other hand, for a corrected correct labeled image generated from a plurality of correct labeled images in which a plurality of annotators, including a less skilled annotator, have selected label regions, the value "0.5" is set to the label region and the value "0" is set to the remaining regions.

[0081] (Variation 2) Finally, a second modification of the machine learning process will be described with reference to Fig. 13. In this modification, the recognition accuracy of the learning model 100 is improved by adjusting the weight of the loss (LOSS) between the confidence image and the correct label.

[0082] Specifically, the loss weight is adjusted by multiplying a loss map indicating the loss between the second confidence image 242 output by the learning model 100 and the second corrected correct-label image 232 by a loss weight map. More specifically, as shown in FIG. 13 , for the second corrected correct-label image 232 including a label region consisting of three regions with different values, the loss between the second confidence image 242 and the second corrected correct-label image 232 is calculated to generate a loss map 350. Meanwhile, a loss weight map 360 is generated in which a value of "0.1" is assigned to a lower region 360a corresponding to the lower label region to which a value of "1 / 3" is assigned in the second corrected correct-label image 232. Here, when the second corrected correct-label image 232 is generated based on correct-label images of three annotators, for example, the region 360a to which the value "0.1" is assigned is a region where only one annotator has selected the label region.

[0083] Then, by multiplying the loss map 350 by the loss weight map 360, a new loss map 370 is generated in which the loss for the region 370a selected by only one annotator is selectively reduced. Then, the parameters of the learning model 100 are updated based on the new loss map 370. This configuration can reduce the influence of mislabeling.

[0084] The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims.

[0085] For example, in the above-described embodiment, multiple experts (annotators) set labels to one ultrasound image 210, thereby generating multiple correctly labeled images. However, the entity that sets labels to ultrasound images is not limited to an operator such as an expert, and labels may be set by, for example, another learning model.

[0086] In the above-described embodiment, the learning model 100 installed in the ultrasound diagnostic device 1 outputs two confidence images. However, the learning model 100 may be configured to output three or more confidence images (estimation results). In this case, three or more corrected correct answer labeled images are generated from multiple correct answer labeled images.

[0087] In the above-described embodiment, the learning model 100 is described as a single multi-task learning model that receives one ultrasound image as input and outputs two confidence images. However, a learning model that receives one ultrasound image as input and outputs two confidence images is not limited to a single learning model. The learning model may be configured with a first learning model that receives one ultrasound image as input and outputs a first confidence image, and a second learning model that receives one ultrasound image as input and outputs a second confidence image. In the field of ultrasound diagnosis, which requires real-time image recognition, a single multi-task learning model is preferable from the standpoint of processing speed.

[0088] In the above-described embodiment, the learning model 100 is trained using ultrasound images as training data, and the trained learning model 100 is installed in the ultrasound diagnostic device 1. However, the learning model 100 may be trained using various medical images, such as X-ray images and MRI images, as training data, and installed in a specialized image diagnostic device. For example, the learning model 100 receives an X-ray image as input and outputs two or more confidence images for a region of interest on the X-ray image. Note that the region of interest on the ultrasound image is not limited to the brachial plexus, and may include various regions such as the radial nerve, ulnar nerve, and median nerve.

[0089] Furthermore, in the above-described embodiment, one display image is generated from two certainty images output by the learning model 100 installed in the ultrasound diagnostic device 1, and is displayed on the display unit 26 of the ultrasound diagnostic device 1. However, the two certainty images output by the learning model 100 may be displayed as they are on the display unit 26 of the ultrasound diagnostic device 1.

[0090] The means and methods for performing various processes in the machine learning device 30 according to the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided by a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred to and stored in a storage unit such as a hard disk drive. The above program may also be provided as standalone application software, or may be incorporated into the software of the machine learning device 30 as a function of the device.

[0091] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims. [Explanation of symbols]

[0092] 1. Ultrasound diagnostic equipment, 10 Ultrasonic transducer, 20 Device body, 30 Machine learning devices, 21,31 Control section, 22,32 memory section, 23,33 Communications Department, 24 Transmitting and receiving unit, 25 ultrasound image generation unit, 26,34 Display section, 27 Operation section, 35 input section, 100 learning models, 200 training data, 210 ultrasound images, 221,222,223,224 correct labeled images, 231,232 corrected correct label images, 241,242 confidence images.

Claims

1. a storage unit that stores a learning model that receives a medical image as input and outputs two or more inference results related to a predetermined region of interest; a generating unit that generates two or more corrected corrected labeled images that are different from each other from a plurality of corrected labeled images that are different from each other and that are generated in advance for the region of interest of one medical image; a learning unit that trains the learning model using the one medical image and the two or more corrected correct-label images as training data; A machine learning device having the above.

2. the learning model receives a medical image as input and outputs a first inference result and a second inference result; the generating unit generates a first corrected correct labeled image and a second corrected correct labeled image from the plurality of correct labeled images; 2. The machine learning device according to claim 1, wherein the learning unit trains the learning model by calculating a loss for a pair of the first inference result and the first corrected correct-labeled image and a pair of the second inference result and the second corrected correct-labeled image.

3. 3. The machine learning device according to claim 1, wherein the two or more modified correct-labeled images include a first modified correct-labeled image and a second modified correct-labeled image having a larger area of ​​a labeled region for the region of interest than the first modified correct-labeled image.

4. The machine learning device according to claim 3 , wherein the label region of the first corrected correct-labeled image and the label region of the second corrected correct-labeled image partially overlap each other.

5. The machine learning device according to claim 1 , wherein the plurality of correctly labeled images includes an image in which a label area is set by an operator.

6. The machine learning device according to claim 1 , wherein the plurality of correctly labeled images include images for which label regions are set by another learning model.

7. The machine learning device according to claim 1 , wherein two or more values ​​are assigned to the corrected correct labeled image.

8. The machine learning device according to claim 7 , wherein the two or more values ​​are changed depending on an operator who sets the label region.

9. 8. The machine learning device according to claim 7, wherein when the learning unit trains the learning model, a weight of a loss between the corrected correct-label image and the inference result is adjusted based on a plurality of correct-label images used to generate the corrected correct-label image.

10. The machine learning device according to claim 1 , wherein the learning model is a single learning model that receives a medical image as input and outputs the two or more inference results.

11. The machine learning device according to claim 1 , wherein the medical image is an ultrasound image.

12. A machine learning method used in a machine learning device having a memory unit that stores a learning model that receives a medical image as input and outputs two or more inference results related to a predetermined region of interest, the method comprising: (a) generating two or more modified corrected labeled images different from each other from a plurality of corrected labeled images different from each other that have been previously generated for the region of interest of one medical image; (b) training the learning model using the one medical image and the two or more corrected correct-label images as training data; A machine learning method having

13. A machine learning program executed in a machine learning device having a storage unit that stores a learning model that receives a medical image as input and outputs two or more inference results related to a predetermined region of interest, a step (a) of generating two or more mutually different corrected correct labeled images from a plurality of mutually different correct labeled images previously generated for the region of interest of one medical image; a step (b) of training the learning model using the one medical image and the two or more corrected correct-label images as training data; a machine learning program that causes the machine learning device to execute the above.

14. a storage unit that stores a learning model that receives a medical image as input and outputs two or more inference results related to a predetermined region of interest; a display unit that displays a display image generated based on the two or more inference results, An image diagnostic device in which the learning model is trained using two or more different corrected correct labels generated from a plurality of different correct labeled images previously generated for the region of interest of a single medical image, and the single medical image as learning data.

15. an ultrasonic probe for transmitting and receiving ultrasonic waves to and from a subject; a storage unit that stores a learning model that receives an ultrasound image acquired by the ultrasound probe as an input and outputs two or more inference results related to a predetermined region of interest; a display unit that displays a display image generated based on the two or more inference results, The learning model is trained using two or more different corrected correct labeled images generated from a plurality of different correct labeled images previously generated for the region of interest of a single ultrasound image, and the single ultrasound image as training data.

Images