Region extraction model creation support device, method of operating the region extraction model creation support device, operating program of the region extraction model creation support device

Abstract

Provided are: a region extraction model creation assistance device that can suppress reduction, as compared with conventional devices, in accuracy of extracting regions of classes of a region extraction model by using local annotation data; a method for operating the region extraction model creation assistance device; and a program for operating the region extraction model creation assistance device. According to the present invention, a CPU of a region extraction model creation assistance server is provided with a learning unit. The learning unit uses, as training data for the region extraction model, local annotation data generated by locally giving labels to training input images and to regions of classes of the training input images. The learning unit causes output, from the region extraction model, of a final feature quantity map having element values relating to probabilities of the regions of the classes. A sharpening process is performed on a probability distribution map that indicates the probabilities for the respective classes and that has been generated on the basis of the final feature quantity map, whereby a post-process probability distribution map is created. The learning unit generates a boundary image on the basis of the post-process probability distribution map that is a result of the sharpening process. The learning unit calculates, as a boundary length loss, an average value of pixel values of the boundary image, and updates the region extraction model in a direction of reducing the average value.

Description

Technical Field

[0001] The technology of the present disclosure relates to an apparatus for assisting in creating a region extraction model, a method of operating the apparatus for assisting in creating a region extraction model, and an operation program of the apparatus for assisting in creating a region extraction model.

Background Art

[0002] In the field of machine learning, for example, region extraction models have been variously developed to extract regions of a certain class in a subject to be recognized, such as recognizing each pulmonary lobe of the lungs shown in a chest tomographic image taken by a CT (Computed Tomography) apparatus in terms of pixels, where regions of a plurality of classes whose boundaries are adjacent to each other are extracted. In such region extraction models, teacher data is required in the learning phase. The teacher data is composed of learning input images and annotation data. The annotation data is generated by manually assigning a label corresponding to a class to the learning input image. In the case of the above example of a chest tomographic image, the annotation data is data generated by assigning labels such as "right upper lobe", "right middle lobe", "left lower lobe", and "outside the lung field" to the chest tomographic image as a learning input image.

[0003] Labels are usually assigned so as to fill the entire region of the class. However, such a method of assigning labels is very laborious. Therefore, in order to reduce the labor of assigning labels, annotation data is generated by roughly assigning labels locally with an interval between regions of the class instead of the entire region of the class, and a technique of training a region extraction model using the thus generated annotation data (hereinafter referred to as local annotation data) has been proposed.

[0004] Using locally annotated data can certainly reduce the effort required for labeling. However, locally annotated data is incomplete as training data compared to annotated data where labels are applied to the entire class region. As a result, the accuracy of class region extraction in the output data from the region extraction model decreases. Specifically, in the output data, class boundaries may appear jagged, or noise indicating class boundaries may be superimposed on areas that are not actually class boundaries.

[0005] Non-patent document 1 addresses the above-mentioned problem of decreased accuracy in extracting class regions in the region extraction model due to the use of locally annotated data by performing the following processing during the training phase of the region extraction model. Specifically, boundary detection processing using a Sobel filter or the like is applied to the probability distribution map (a map showing the probability of each class being a region) preceding the output data to generate boundary images from the probability distribution map. Then, the average value of the pixel values ​​of the boundary images is incorporated into the loss of the region extraction model, and the region extraction model is updated in a way that reduces the loss. Reducing the loss by incorporating the average value of the pixel values ​​of the boundary images leads to smoothing out the jaggedness of class boundaries in the output data using the probability distribution map, or to removing noise that indicates class boundaries that lie on parts that are not actually class boundaries. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Mehran Javanmardi, etc., “Unsupervised Total Variation Loss for Semi-supervised Deep Learning of Semantic Segmentation”, ECCV-16(submission ID 868), 4 May 2016. [Overview of the project] [Problems that the invention aims to solve]

[0007] In Non-Patent Document 1, if the boundary image generated from the probability distribution map is not a binary image of, for example, 0 and 1, but rather an image that can take any value between 0 and 255, then undesirable learning may occur, as shown in Figure 28 as an example.

[0008] In Figure 28, for simplicity, consider a star-shaped class region 200. In this case, it is desirable that learning be performed to smooth the jagged edges of the class boundary in the probability distribution map, as shown in the upper circle 201. However, there is a risk that undesirable learning will be performed, as shown in the lower star shape 202, which smooths the change in class values ​​in the probability distribution map. This is because smoothing the jagged edges of the class boundary in the probability distribution map (taking off the corners of the star shape of region 200 to make circle 201) and smoothing the change in class values ​​in the probability distribution map (maintaining the star shape of region 200 while gradually decreasing the values ​​from the center of region 200 towards the boundary) are equivalent in that they reduce the loss (in this case, the average value of the pixel values ​​of the boundary image).

[0009] If the learning process unintentionally smooths out the changes in class values ​​in the probability distribution map, it will naturally suppress the smoothing of the jagged edges of class boundaries in the probability distribution map. Therefore, the problem of reduced accuracy in extracting class regions in region extraction models due to the use of locally annotated data still remains.

[0010] One embodiment of the technology of this disclosure provides a region extraction model creation support device, an operating method for the region extraction model creation support device, and an operating program for the region extraction model creation support device that can suppress the decrease in the accuracy of extracting class regions of a region extraction model by using local annotation data more effectively than in the conventional method. [Means for solving the problem]

[0011] The region extraction model creation support device of this disclosure is a region extraction model creation support device that assists in creating a region extraction model that extracts regions of multiple classes whose boundaries are contiguous with each other, which are regions of a class in an object to be recognized that is captured in an image, and comprises a processor and memory connected to or built into the processor, wherein the processor uses a training input image and local annotation data generated by locally labeling the regions of the classes in the training input image as training data, causes the region extraction model to output a final feature map having element values ​​related to the probability of a region being of a class, applies a sharpening process to the final feature map or a probability distribution map that shows the probability for each class, which is generated based on the final feature map, detects boundaries based on the results of the sharpening process, and updates the region extraction model in a direction that reduces the boundary length loss corresponding to the length of the boundaries.

[0012] Preferably, the processor outputs training output data from the region extraction model, which is obtained by extracting regions of a class from the training input image. It then calculates the loss of the region extraction model based on the comparison result between the local annotation data for the labeled local areas and the training output data. The loss and boundary length loss are added together to obtain the first overall loss, and the region extraction model is updated in a direction that reduces the first overall loss.

[0013] The processor preferably updates the region extraction model in a direction that reduces the second overall loss by adding the size loss corresponding to the size of multiple classes of regions to the first overall loss. This second overall loss is obtained by further adding the size loss to the first overall loss.

[0014] The sharpening process is preferably a process that applies a temperature-sensitive softmax function with a temperature parameter of 1 or less to the final feature map or probability distribution map.

[0015] The sharpening process is preferably a process that applies the softargmax function to the final feature map or probability distribution map.

[0016] The sharpening process is preferably a process that applies a sigmoid function with a gain of 1 or more to the final feature map or probability distribution map.

[0017] The boundary length loss is preferably the average value of the pixel values ​​in the boundary image generated by detecting the boundary from the result of the sharpening process.

[0018] The processor preferably accepts the specification of the region to be detected as a boundary in the result of the sharpening process.

[0019] The image should preferably be a medical image.

[0020] The class preferably includes lung lobes.

[0021] The method for operating the region extraction model creation support device of this disclosure is a method for operating the region extraction model creation support device that supports the creation of a region extraction model that extracts regions of multiple classes whose boundaries are contiguous with each other, which are regions of a class in an object to be recognized that is captured in an image, and includes using a training input image and local annotation data generated by locally labeling the regions of the classes in the training input image as training data, outputting a final feature map from the region extraction model that has element values ​​related to the probability of a region being of a class, applying a sharpening process to the final feature map, or a probability distribution map that shows the probability for each class and is generated based on the final feature map, detecting boundaries based on the results of the sharpening process, and updating the region extraction model in a direction that reduces the boundary length loss according to the length of the boundaries.

[0022] The operation program of the region extraction model creation support device of the present disclosure is an operation program of a region extraction model creation support device that supports the creation of a region extraction model for extracting regions of a class in a subject to be recognized shown in an image, where the regions of a plurality of classes whose boundaries are adjacent to each other are extracted, and uses, as teacher data, a learning input image and local annotation data generated by locally labeling the regions of the class of the learning input image, causes the region extraction model to output a final feature map having an element value related to the probability of being a region of a class, performs a sharpening process on the final feature map or a probability distribution map showing probabilities for each class generated based on the final feature map, detects a boundary based on the result of the sharpening process, and updates the region extraction model in a direction to reduce a boundary length loss according to the length of the boundary, and includes Let the computer perform the process. 。

Advantages of the Invention

[0023] According to the technology of the present disclosure, it is possible to provide a region extraction model creation support device, an operation method of the region extraction model creation support device, and an operation program of the region extraction model creation support device that can suppress a decrease in the extraction accuracy of the regions of the class of the region extraction model due to the use of local annotation data more effectively than in the prior art.

Brief Description of the Drawings

[0024] [Figure 1] It is a diagram showing a region extraction model creation support server and an annotator terminal. [Figure 2] It is a diagram showing a learning input image and local annotation data. [Figure 3] It is a diagram showing a state of transmitting teacher data from an annotator terminal to a region extraction model creation support server. [Figure 4] It is a block diagram showing a computer constituting a region extraction model creation support server. [Figure 5] It is a block diagram showing a processing unit of a CPU of a region extraction model creation support server. [Figure 6]This is a block diagram showing the detailed configuration of the region extraction model. [Figure 7] This diagram shows the elements of a probability distribution map. [Figure 8] This is a diagram showing the sharpening process. [Figure 9] This diagram shows the details of the sharpening process. [Figure 10] This is a diagram showing the boundary detection process. [Figure 11] This figure shows the process for calculating the average value. [Figure 12] This diagram shows the loss calculation process. [Figure 13] This diagram shows the update settings process. [Figure 14] This figure shows an example of how boundary images change from the beginning to the end of the learning process. [Figure 15] This is a flowchart showing the processing procedure of the region extraction model creation support server. [Figure 16] This figure shows how an input image with unknown regions for each class is input into a trained region extraction model, and how the trained region extraction model outputs output data with labels assigned to the regions of each class in the input image. [Figure 17] This figure shows the process for calculating the variance of lung lobe size. [Figure 18] This figure shows the update setting process of the second embodiment. [Figure 19] This figure shows the training output data when the model gets stuck in a local minimum. [Figure 20] This figure shows the local annotation data for the third embodiment. [Figure 21] This diagram shows the process of inputting training images into a lung field extraction model and outputting lung field extraction data from the lung field extraction model. [Figure 22] This figure shows the boundary detection process of the third embodiment. [Figure 23] This figure shows the sharpening process according to the fourth embodiment. [Figure 24] This figure shows the sharpening process according to the fifth embodiment. [Figure 25]This figure shows another example of the sharpening process according to the fifth embodiment. [Figure 26] This figure shows the sharpening process according to the sixth embodiment. [Figure 27] This figure shows another example of the sharpening process according to the sixth embodiment. [Figure 28] This diagram illustrates the problems with conventional technology. [Modes for carrying out the invention]

[0025] [First Embodiment] As an example, as shown in Figure 1, the region extraction model creation support server (hereinafter referred to as the support server) 10 is connected to the annotator terminal 12 via the network 11, enabling mutual communication. The network 11 is a WAN (Wide Area Network), such as the Internet or a public communication network. Although only one annotator terminal 12 is depicted, there are actually multiple terminals.

[0026] The support server 10 is, for example, a server computer, a workstation, etc. The support server 10 assists in the creation of a region extraction model 41 (see Figure 5) that extracts regions of multiple classes whose boundaries are contiguous and which are regions of a class in the object to be recognized that is captured in the image. In other words, the support server 10 is an example of a "region extraction model creation support device" related to the technology of this disclosure. The annotator terminal 12 is a terminal operated by an annotator that generates local annotation data 16 (see Figure 2). The annotator terminal 12 is, for example, a personal computer, a tablet terminal, etc.

[0027] As an example, as shown in Figure 2, the annotator terminal 12 displays the training input image 15L on its display. In this example, the training input image 15L is a tomographic image obtained by taking a CT (Computed Tomography) scan of the chest of a patient suspected of having a lung disease such as pneumonia or lung cancer. As is well known, a CT scanner takes radiographic images of a patient at different projection angles to acquire multiple projection data, and outputs a tomographic image by reconstructing the acquired multiple projection data. A tomographic image is voxel data that represents the three-dimensional shape of the internal structure of the patient. Figure 2 shows a tomographic image at a slice position in an axial section. In this example, the tomographic image is mainly an image of the patient's lungs. The training input image 15L is an example of a "medical image" related to the technology of this disclosure.

[0028] The annotator operates the input device of the annotator terminal 12 to locally assign labels LB, which are squares with a size of several pixels, to the regions of each class in the training input image 15L. Specifically, the annotator assigns label LB1 to the region thought to be the right upper lobe, label LB2 to the region thought to be the right middle lobe, and label LB3 to the region thought to be the right lower lobe. The annotator also assigns label LB4 to the region thought to be the left upper lobe and label LB5 to the region thought to be the left lower lobe. Furthermore, the annotator assigns label LB6 to the region thought to be outside the lung field. This generates local annotation data 16. As can be seen from this explanation, the classes in this example are the lung lobes (right upper lobe, right middle lobe, right lower lobe, left upper lobe, and left lower lobe) and the region outside the lung field. In other words, the classes include the lung lobes. The annotator then assigns a label LB to each slice position in the tomographic image to generate local annotation data 16.

[0029] Note that in Figure 2, human body structures are drawn on the local annotation data 16 for the purpose of aiding understanding, but the actual local annotation data 16 does not contain data on human body structures, but only data on the assigned labels LB. More specifically, the local annotation data 16 is data in which pairs of label LB type and the position coordinates of pixels in the training input image 15L to which label LB is assigned are registered.

[0030] As an example, as shown in Figure 3, the annotator terminal 12 sends a set of training input images 15L and local annotation data 16 to the support server 10 as training data 20. The support server 10 receives the training data 20 from the annotator terminal 12. The support server 10 uses the received training data 20 to train the region extraction model 41.

[0031] As an example, as shown in Figure 4, the computer comprising the support server 10 includes storage 30, memory 31, CPU (Central Processing Unit) 32, communication unit 33, display 34, and input device 35. These are interconnected via a bus line 36.

[0032] Storage 30 is a hard disk drive built into the computer that makes up the support server 10, or connected via cable or network. Alternatively, storage 30 is a disk array consisting of multiple hard disk drives installed in series. Storage 30 stores control programs such as the operating system, various application programs, and various data associated with these programs. A solid-state drive may be used instead of a hard disk drive.

[0033] Memory 31 is work memory for the CPU 32 to execute processing. The CPU 32 loads the program stored in storage 30 into memory 31 and executes processing according to the program. In this way, the CPU 32 comprehensively controls each part of the computer. CPU 32 is an example of a "processor" related to the technology of this disclosure. Note that memory 31 may be built into the CPU 32.

[0034] The communication unit 33 is a network interface that controls the transmission of various types of information via the network 11, etc. The display 34 displays various screens. These screens are equipped with GUI (Graphical User Interface) operation functions. The computers constituting the support server 10 receive operation instructions from input devices 35 through the various screens. The input devices 35 include keyboards, mice, and touch panels.

[0035] As an example, as shown in Figure 5, the storage 30 of the support server 10 stores an operating program 40. The operating program 40 is an application program that causes the computers constituting the support server 10 to function as a "region extraction model creation support device" according to the technology of this disclosure. In other words, the operating program 40 is an example of an "operating program for a region extraction model creation support device" according to the technology of this disclosure. In addition to the operating program 40, the storage 30 stores multiple training data 20 from the annotator terminal 12, a region extraction model 41, a temperature-sensitive softmax function 42, and a boundary detection filter 43.

[0036] When the operating program 40 is started, the CPU 32 of the support server 10 works in cooperation with the memory 31 and other components to function as a read / write (hereinafter abbreviated as RW) control unit 50 and a learning unit 51.

[0037] The RW control unit 50 controls the storage of various data to the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 50 stores the training data 20 from the annotator terminal 12 in the storage 30. The RW control unit 50 also reads the region extraction model 41, the temperature-sensitive softmax function 42, and the boundary detection filter 43 from the storage 30 and outputs the read region extraction model 41, the temperature-sensitive softmax function 42, and the boundary detection filter 43 to the learning unit 51. The RW control unit 50 also reads the training data 20 from the storage 30 and outputs the read training data 20 to the learning unit 51. Furthermore, the RW control unit 50 stores the trained region extraction model (hereinafter referred to as the trained region extraction model) 41LD (see Figure 16) from the learning unit 51 in the storage 30.

[0038] The region extraction model 41 identifies each of the lungs mentioned above. leaf This is a machine learning model for extracting regions outside the lung field. The region extraction model 41 is constructed using a convolutional neural network (CNN), such as ResNet (Residual Networks) or U-Net (U-shaped Networks). The learning unit 51 trains the region extraction model 41 using the training data 20, a temperature-sensitive softmax function 42, and a boundary detection filter 43.

[0039] As an example, as shown in Figure 6, the learning unit 51 inputs the training input image 15L from the training data 20 to the region extraction model 41. The region extraction model 41 consists of an encoder unit 60 and an output unit 61. The encoder unit 60 has multiple convolutional layers that perform convolution processing using filters, and multiple pooling layers that perform pooling processing to reduce the data after convolution by obtaining local statistics of the data after convolution. The encoder unit 60 converts the training input image 15L into a feature map 62. The encoder unit 60 outputs the feature map 62 to the output unit 61. Although not shown in the figure, the encoder unit 60 also performs skip layer processing, etc., to pass the data after convolution to the output unit 61.

[0040] The output unit 61 includes a decoder unit 63, a probability distribution map generation unit 64, and a labeling unit 65. The decoder unit 63 performs an upsampling process to enlarge the feature map 62 into an enlarged feature map. The decoder unit 63 also performs a convolution process simultaneously with the upsampling process. Furthermore, the decoder unit 63 performs a merge process to combine the enlarged feature map with the data after the convolution process passed from the encoder unit 60 in the skip layer process. The decoder unit 63 performs another convolution process after the merge process. Through these various processes, the decoder unit 63 converts the feature map 62 into the final feature map 66.

[0041] The final feature map 66, also known as logits, has elements that correspond one-to-one with the pixels of the training input image 15L. Each element of the final feature map 66 has an element value associated with each class. The decoder unit 63 outputs the final feature map 66 to the probability distribution map generation unit 64.

[0042] The probability distribution map generation unit 64 generates a probability distribution map 67 from the final feature map 66 using a well-known activation function such as the softmax function. The probability distribution map generation unit 64 outputs the probability distribution map 67 to the labeling unit 65.

[0043] As an example, as shown in Figure 7, the probability distribution map 67, like the final feature map 66, has elements 70 that correspond one-to-one with the pixels of the training input image 15L, and the element value of each element 70 is the registered probability that it is a region of each class. In Figure 7, an example is shown where the probability that the element 70 is a region of the upper right lobe is 86% (0.86), and the probabilities that it is a region of the right middle lobe, right lower lobe, left upper lobe, left lower lobe, and outside the lung field are 6%, 1%, 2%, 3%, and 2%, respectively. The probabilities of being a region of each class are all added together to 100%.

[0044] The labeling unit 65 assigns the label LB of the class with the highest probability to each element 70 of the probability distribution map 67. For example, in the example shown in Figure 7, the labeling unit 65 assigns label LB1 to the upper right leaf, which has the highest probability of 86%. By assigning label LB to each element 70 of the probability distribution map 67 in this way, the labeling unit 65 outputs training output data 68L. The training output data 68L is data in which one of the six classes of label LB1 to LB6 is assigned to each pixel of the training input image 15L.

[0045] As an example, as shown in Figure 8, the learning unit 51 applies a sharpening process 75 to the probability distribution map 67 output from the region extraction model 41, and the processed probability distribution map 67P is obtained. The sharpening process 75 is a process that applies a temperature-sensitive softmax function 42 to the probability distribution map 67. The temperature-sensitive softmax function 42 is the function shown in the callout in Figure 8 and in equation (1) below. Here, x is the probability of each class in the probability distribution map 67. Also, i and j represent the class numbers, and i represents the class number for which the probability is to be calculated. Also, T is the temperature parameter, and a sufficiently small value of 1 or less (T≦1) is set.

[0046]

number

[0047] As an example, as shown in Figure 9, the probabilities of each class element 70 of the probability distribution map 67 are transformed by applying the temperature-controlled softmax function 42 to element 70P of the processed probability distribution map 67P, such that, for example, the probability of one class becomes 99.999% (0.99999) and the probabilities of all other classes become 0% (0.00000). In other words, while the probability distribution map 67 is data that can take any value between 0 and 1, the processed probability distribution map 67P is data that can take both extreme values, such as binary values ​​of 0.99999 and 0.00000. The processed probability distribution map 67P is an example of the "result of sharpening processing" related to the technology of this disclosure.

[0048] As an example, as shown in Figure 10, the learning unit 51 applies a boundary detection filter 43 to the processed probability distribution map 67P to perform boundary detection processing 80 and generate a boundary image 81. The boundary detection filter 43 is, for example, a Previtt filter or a Sobel filter. The boundary image 81 is an image in which the pixel values ​​of pixels in the parts that appear to be the boundaries of each class region are remarkably large. Note that although the boundary image 81 is depicted as a two-dimensional image in Figure 10, the actual boundary image 81 is a three-dimensional image. The same applies to Figure 14 and the following.

[0049] As an example, as shown in Figure 11, the learning unit 51 performs an average value calculation process 85. The average value calculation process 85 is a process that calculates the average value 86 of the pixel values ​​of the boundary image 81. The average value 86 is larger the more areas in the boundary image 81 that appear to be boundaries. Therefore, if the areas that appear to be class boundaries are jagged, or if there is noise in areas that are not actually class boundaries, the average value 86 will be large. The average value 86 is an example of the "boundary length loss" related to the technology of this disclosure.

[0050] As an example, as shown in Figure 12, the learning unit 51 performs a loss calculation process 90. The loss calculation process 90 is a process that calculates the loss 91 of the region extraction model 41 according to the comparison result between the local annotation data 16 and the training output data 68L. The loss 91 is the cross-entropy error. In the loss calculation process 90, the loss 91 is calculated only for the locals to which labels LB have been assigned. If the labels LB assigned to the local annotation data 16 and the labels LB assigned to the training output data 68L are different, the loss 91 will be large. Conversely, if the labels LB assigned to the local annotation data 16 and the labels LB assigned to the training output data 68L are the same, the loss 91 will be small.

[0051] As an example, as shown in Figure 13, the learning unit 51 calculates the first overall loss 95 by adding the loss 91 of the region extraction model 41 and the average value 86 as the boundary length loss. Then, the learning unit 51 performs an update setting process 96. The update setting process 96 is a process that updates the values ​​of the filter coefficients of the region extraction model 41 in a way that reduces the first overall loss 95 using the well-known backpropagation method. This update setting process 96 improves the extraction accuracy of the region extraction model 41 compared to before.

[0052] Figure 14 shows an example of the changes in boundary image 81 from the beginning to the end of training. In the early stages of training, boundary image 81 has a lot of jaggedness in the areas that appear to be class boundaries, and noise is superimposed on areas that are not actually class boundaries. However, as training progresses, the jaggedness in the areas that appear to be class boundaries is reduced, and the noise is also removed. Ultimately, in boundary image 81 at the end of training, the jaggedness in the areas that appear to be class boundaries is almost completely gone, and the noise has also been almost completely removed.

[0053] Next, the operation of the above configuration will be explained by referring to the flowchart shown in Figure 15 as an example. When the operating program 40 is started, the CPU 32 of the support server 10 functions as the RW control unit 50 and the learning unit 51, as shown in Figure 5.

[0054] The RW control unit 50 reads the region extraction model 41, the temperature-sensitive softmax function 42, and the boundary detection filter 43 from the storage 30 and outputs them to the learning unit 51. The RW control unit 50 also reads one training data 20 from the storage 30 and outputs it to the learning unit 51.

[0055] In the learning unit 51, as shown in Figure 6, the training input image 15L is input to the region extraction model 41. As a result, the probability distribution map 67 and the training output data 68L are output from the region extraction model 41 (step ST100).

[0056] As shown in Figure 8, a temperature-sensitive softmax function 42 is applied to the probability distribution map 67 to perform sharpening processing 75, and the probability distribution map 67 is converted to the processed probability distribution map 67P (step ST110). Subsequently, as shown in Figure 10, a boundary detection filter 43 is applied to the processed probability distribution map 67P to perform boundary detection processing 80, and a boundary image 81 is generated (step ST120).

[0057] As shown in Figure 11, an average value calculation process 85 is performed, and the average value 86 of the pixel values ​​of the boundary image 81 is calculated as the boundary length loss (step ST130). Also, as shown in Figure 12, a loss calculation process 90 is performed, and the loss 91 of the region extraction model 41 is calculated according to the comparison result between the local annotation data 16 for the local area with label LB and the training output data 68L (step ST140).

[0058] As shown in Figure 13, the update setting process 96 is performed, and the region extraction model 41 is updated in a way that reduces the first total loss 95, which is the sum of the loss 91 of the region extraction model 41 and the average value 86 (step ST150). This series of processes from steps ST100 to ST150 is repeated as long as the extraction accuracy of the region extraction model 41 does not exceed a predetermined threshold (NO in step ST160).

[0059] If the extraction accuracy of the region extraction model 41 exceeds a threshold (YES in step ST160), the region extraction model 41 is stored in the storage 30 by the RW control unit 50 as a trained region extraction model 41LD.

[0060] As an example, the trained region extraction model 41LD receives an input image 15 in which the regions of each class are unknown, as shown in Figure 16. The trained region extraction model 41LD outputs output data 68 in which the regions of each class in the input image 15 are labeled LB. In the output data 68, the boundaries of the regions of each class are contiguous.

[0061] As described above, the CPU 32 of the support server 10 is equipped with a learning unit 51. The learning unit 51 uses the learning input image 15L and the local annotation data 16 generated by locally assigning labels LB to the class regions of the learning input image 15L as training data 20 for the region extraction model 41. The learning unit 51 causes the region extraction model 41 to output a final feature map 66 having element values ​​related to the probability of class regions. Then, a sharpening process 75 is applied to the probability distribution map 67 that shows the probability for each class, which is generated based on the final feature map 66, to obtain a processed probability distribution map 67P. The learning unit 51 generates a boundary image 81 based on the processed probability distribution map 67P, which is the result of the sharpening process. The learning unit 51 calculates the average value 86 of the pixel values ​​of the boundary image 81 as the boundary length loss and updates the region extraction model 41 in a direction that reduces the average value 86.

[0062] The processed probability distribution map 67P is data that takes on both extreme values, such as 0.99999 and 0.00000. Therefore, the boundary image 81 generated based on the processed probability distribution map 67P is not an image that takes on any value between 0 and 255, for example, but an image that takes on only a few values. Accordingly, the risk of falling into undesirable learning that smooths out the changes in class values ​​in the probability distribution map 67, as described in Non-Patent Literature 1, is suppressed. As a result of the above, the technology of this disclosure makes it possible to suppress the decrease in the accuracy of extracting class regions of the region extraction model 41 by using local annotation data 16 more than in the conventional method.

[0063] Reducing the average boundary length loss of 86 means smoothing out the jaggedness of class boundaries in the probability distribution map 67 and, consequently, in the training output data 68L, and removing noise that indicates class boundaries but is actually present in areas that are not class boundaries. As a result, as shown in Figure 14, towards the end of training, the jaggedness of areas that appear to be class boundaries is almost eliminated, and the noise is also almost removed, allowing for more accurate extraction of class regions.

[0064] The learning unit 51 outputs training output data 68L from the region extraction model 41, which is obtained by extracting regions of a specific class from the training input image 15L. The learning unit 51 then calculates the loss 91 of the region extraction model 41 based on the comparison result between the local annotation data 16 for the local areas to which labels LB have been assigned and the training output data 68L. The learning unit 51 adds the loss 91 and the average value 86 as boundary length loss to obtain the first overall loss 95, and updates the region extraction model 41 in a direction that reduces the first overall loss 95.

[0065] By updating the region extraction model 41 in a way that reduces the first overall loss 95, which is obtained by incorporating the average value 86 as a boundary length loss into the loss 91 of the region extraction model 41, the learning process of removing jaggedness from parts that appear to be class boundaries and removing noise can be performed as part of the learning process to reduce the loss 91 of the region extraction model 41.

[0066] The sharpening process 75 applies a temperature-sensitive softmax function 42 to the probability distribution map 67. This allows the probability distribution map 67 to be easily converted into the processed probability distribution map 67P. Furthermore, it enables the update setting process 96 using the backpropagation method to be performed smoothly.

[0067] The boundary length loss is the average value 86 of the pixel values ​​of the boundary image 81, which is generated by detecting the boundary from the processed probability distribution map 67P. Therefore, the boundary length loss can be easily calculated. Alternatively, instead of the average value 86, the sum of the pixel values ​​of the boundary image 81 may be used as the boundary length loss.

[0068] In this example, tomographic images of the patient's lungs are primarily used as the training input image 15L. Furthermore, the classes in this example include the lung lobes (right upper lobe, right middle lobe, right lower lobe, left upper lobe, and left lower lobe) and the area outside the lung field, thus encompassing the lung lobes.

[0069] In the medical field, there is a strong demand to support physicians' diagnoses by extracting regions of organs from medical images using a region extraction model 41 and presenting the results to physicians. Furthermore, lung diseases such as pneumonia, which has been increasing as a cause of death among the elderly in recent years, and lung cancer, which is the leading cause of cancer death among men, are receiving particular attention. Therefore, there is a strong demand to recognize lung lobes related to lung diseases with a certain degree of accuracy without requiring much effort. The technology disclosed herein can meet these demands because it uses medical images and includes lung lobes in its classes.

[0070] [Second Embodiment] As an example, as shown in Figure 17, in this embodiment, the learning unit 51 performs a lobe size variance calculation process 100 to calculate the variance of the size of each lung lobe region (hereinafter referred to as lobe size variance) 101. First, the learning unit 51 obtains the total number of pixels to which each lung lobe label LB1 to LB5 is assigned from the learning output data 68L. The total number of pixels to which each lung lobe label LB1 to LB5 is assigned represents the size of each lung lobe region. The learning unit 51 calculates the lobe size variance 101 from the obtained total number of pixels. The lobe size variance 101 takes on a larger value as the size of each lung lobe varies. The lobe size variance 101 is an example of "size loss" related to the technology of this disclosure.

[0071] As an example, as shown in Figure 18, the learning unit 51 calculates a second overall loss 105 by summing the loss 91 of the region extraction model 41, the average value 86 as boundary length loss, and the lobe size variance 101 as size loss. The second overall loss 105 is obtained by adding the lobe size variance 101 as size loss to the first overall loss 95 of the first embodiment described above, which is the sum of the loss 91 of the region extraction model 41 and the average value 86 as boundary length loss.

[0072] The learning unit 51 performs an update setting process 106. Similar to the update setting process 96 in the first embodiment described above, the update setting process 106 is a process that updates the values ​​of the filter coefficients of the region extraction model 41 in a direction that reduces the second overall loss 105 using the well-known backpropagation method. This update setting process 106 improves the extraction accuracy of the region extraction model 41 compared to before.

[0073] If the update setting process 96, which updates the region extraction model 41 in a direction that reduces the average value 86 as boundary length loss, is excessively applied, there is a concern that it may fall into a local minimum as shown in Figure 19, for example. That is, there is a risk that training output data 68L with label LB5 will be output only for regions in the lower left leaf that have been assigned label LB5 in the local annotation data 16. This is because the regions in this training output data 68L that have been assigned label LB5 have short boundary lengths, i.e., a small average value 86. Also, since the regions in the training output data 68L that have been assigned label LB5 are almost the same as the regions in the local annotation data 16 that have been assigned label LB5, the loss 91 of the region extraction model 41 will also be small. Therefore, even if such training output data 68L is output, it may be mistakenly assumed that the training is going well.

[0074] Since it is known that each lung lobe is roughly the same size, the local optima shown in Figure 19 are not realistically possible, but there is a concern that the system may fall into the local optima shown in Figure 19 for the reasons mentioned above. Therefore, in the second embodiment, the learning unit 51 further adds the lobe size variance 101, which is a size loss corresponding to the size of the region of each lung lobe, to the first total loss 95 to obtain a second total loss 105. The learning unit 51 then updates the region extraction model 41 in a direction that reduces the second total loss 105.

[0075] Reducing the second overall loss of 105 means reducing the lobe size variance of 101. Reducing the lobe size variance of 101 leads to the size of each lobe being approximately the same in the training output data 68L. In other words, it eliminates the concern of falling into a local minimum, as shown in the training output data 68L in Figure 19. Alternatively, the standard deviation of the size of each lobe may be used as the size loss instead of the lobe size variance of 101.

[0076] [Third Embodiment] As an example, as shown in Figure 20, in this embodiment, local annotation data 110 is used in which labels LB1 to LB5 are assigned to each lung lobe, but the label LB6 outside the lung field is not assigned.

[0077] As an example, as shown in Figure 21, the learning unit 51 inputs the training input image 15L to the lung field extraction model 112 prior to training the region extraction model 41. The lung field extraction model 112 is a pre-trained machine learning model prepared separately from the region extraction model 41. The lung field extraction model 112 extracts the lung field regions shown in the training input image 15L and outputs lung field extraction data 113 (see also Figure 22) with labels assigned to the extracted lung field regions.

[0078] As an example, as shown in Figure 22, the processed probability distribution map 115P of this embodiment uses local annotation data 110 that does not have labels LB6 for areas outside the lung field. Therefore, the data distinguishes between individual lung lobes but not between areas outside the lung field. The learning unit 51 applies a boundary detection filter 43 to this processed probability distribution map 115P and performs boundary detection processing 116 to generate a boundary image 117.

[0079] When generating the boundary image 117, the learning unit 51 uses the lung field region represented by the lung field extraction data 113 as the boundary detection target region. In other words, the learning unit 51 accepts the specification of the boundary detection target region in the processed probability distribution map 115P by outputting the lung field extraction data 113 from the lung field extraction model 112. This reduces the processing load of the boundary detection process 116 and shortens the processing time of the boundary detection process 116 compared to the first embodiment in which the boundary detection process 80 is applied to the entire processed probability distribution map 67P, including areas outside the lung field. Furthermore, it is not necessary to assign labels LB6 to areas outside the lung field, further reducing the effort required to assign labels LB. Note that the user may also specify the boundary detection target region in the processed probability distribution map 115P.

[0080] [Fourth Embodiment] In the first embodiment described above, the sharpening process 75 is exemplified by applying a temperature-sensitive softmax function 42 to the probability distribution map 67, but it is not limited to this. As an example, the sharpening process 120 shown in Figure 23 may be performed.

[0081] In Figure 23, the sharpening process 120 of this embodiment is performed in the probability distribution map generation unit 64 of the region extraction model 41. The probability distribution map generation unit 64 applies the sharpening process 120 to the final feature map 66, making the final feature map 66 the probability distribution map 67. The sharpening process 120 is a process that applies the temperature-sensitive softmax function 42 to the final feature map 66, not to the probability distribution map 67.

[0082] In this case as well, the probabilities of each class of element 70 in the probability distribution map 67 will be data that takes on two extreme values, such as 0.99999 and 0.00000, similar to the processed probability distribution map 67P of the first embodiment described above. In this embodiment, the probability distribution map 67 is an example of the "result of sharpening processing" related to the technology of this disclosure.

[0083] [Fifth Embodiment] The sharpening process is not limited to applying a temperature-controlled softmax function 42. For example, the sharpening process 125 shown in Figure 24, or the sharpening process 130 shown in Figure 25, may be performed.

[0084] In Figure 24, the sharpening process 125 is a process that applies the softargmax function 126 to the final feature map 66, and outputs the final feature map 66 as output data 68. The softargmax function 126 is the function shown in the callout in Figure 24 and in equation (2) below. Note that x is an element value related to the probability of each class in the final feature map 66. Also, β is a coefficient and is set to a sufficiently large value. Also, i and j represent the class numbers.

[0085]

number

[0086] The main part of the softargmax function 126 is the same as the temperature-sensitive softmax function 42, except that 1 / T is replaced with β. That is, the softargmax function 126 is the sum of the outputs of each class in the temperature-sensitive softmax function 42 multiplied by the number i representing each class. For example, if in an element of the final feature map 66 the output of class 2 out of the four classes 1 to 4 is 0.99999, and the outputs of the other classes 1, 3, and 4 are 0.00000, then the solution of the softargmax function 126 is 0.00000 × 1 + 0.99999 × 2 + 0.00000 × 3 + 0.00000 × 4 = 1.99998 ≈ 2. In other words, the solution of the softargmax function 126 is approximately the same as the number representing the class. For this reason, the label LB can also be assigned in the sharpening process 125 using the softargmax function 126.

[0087] The sharpening process 130 shown in Figure 25 is a process that applies the softargmax function 126 to the probability distribution map 67, not the final feature map 66. In this case, x in the softargmax function 126 is the probability of each class in the probability distribution map 67. In this embodiment, the output data 68 is an example of the "results of the sharpening process" related to the technology of this disclosure.

[0088] [Sixth Embodiment] Furthermore, if there are two classes, the sharpening process 135 shown in Figure 26 as an example, or the sharpening process 140 shown in Figure 27 as an example, may be performed.

[0089] In Figure 26, the sharpening process 135 is performed in the probability distribution map generation unit 64 of the region extraction model 41, just as in the sharpening process 120 of the fourth embodiment described above. The probability distribution map generation unit 64 applies the sharpening process 135 to the final feature map 66, making the final feature map 66 the probability distribution map 67. The sharpening process 135 is a process that applies the sigmoid function 136A to the final feature map 66. The sigmoid function 136A is the function shown in the callout in Figure 26 and in equation (3) below. Here, x is an element value related to the probability of each class in the final feature map 66. Also, α is the gain, and a sufficiently large value of 1 or more (α≧1) is set.

[0090]

number

[0091] In this case as well, the probabilities of each class of element 70 in the probability distribution map 67 will be data that takes on two extreme values, such as 0.99999 and 0.00000, similar to the processed probability distribution map 67P of the first embodiment described above. In Figure 26, as in the fourth embodiment described above, the probability distribution map 67 is an example of the "result of sharpening processing" related to the technology of this disclosure.

[0092] The sharpening process 140 shown in Figure 27 applies the sigmoid function 136B to the probability distribution map 67, not the final feature map 66, to obtain the processed probability distribution map 67P. The sigmoid function 136B is the function shown in the callout in Figure 27 and in equation (4) below. Here, x is the probability of each class in the probability distribution map 67. Also, α is the gain, and a sufficiently large value of 1 or greater (α≧1) is set. In Figure 27, as in the first embodiment described above, the processed probability distribution map 67P is an example of the "result of the sharpening process" related to the technology of this disclosure.

[0093]

number

[0094] Thus, the sharpening process may also be a process 135 or 140 in which a sigmoid function 136A or 136B with a gain α of 1 or more is applied to the final feature map 66 or probability distribution map 67.

[0095] For example, a cross-sectional image of the pancreas may be used, and the pancreatic head, body, and tail may be extracted as the class region. Alternatively, images taken by street surveillance cameras may be used, and cars, motorcycles, bicycles, and pedestrians may be extracted as the class region. As these examples show, the images are not limited to medical images, and the class does not necessarily have to include the lung field.

[0096] The hardware configuration of the computers constituting the support server 10 can be modified in various ways. For example, the support server 10 can be composed of multiple computers separated as hardware, in order to improve processing power and reliability. For example, the functions of sharpening processing 75 and boundary detection processing 80 of the learning unit 51, and the functions of average value calculation processing 85, loss calculation processing 90, and update setting processing 96 can be distributed among two computers. In this case, the support server 10 is composed of two computers.

[0097] Thus, the hardware configuration of the support server 10 computer can be appropriately changed according to the required performance, such as processing power, security, and reliability. Furthermore, not only the hardware, but also application programs such as the operating program 40 can, of course, be duplicated or distributed and stored on multiple storage devices for the purpose of ensuring security and reliability.

[0098] In each of the above embodiments, the hardware structure of the Processing Unit that performs various processes, such as the RW control unit 50 and the learning unit 51, can be the following types of processors. These types of processors include a CPU 32, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, as well as a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing, and / or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration specifically designed to perform a particular process.

[0099] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0100] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0101] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor elements.

[0102] The technology of this disclosure can be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is understood that various configurations can be adopted without departing from the spirit of the invention, and the invention is not limited to the embodiments described above. Moreover, the technology of this disclosure extends not only to programs but also to storage media for storing programs non-temporarily.

[0103] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0104] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0105] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference. [Explanation of symbols]

[0106] 10. Region Extraction Model Creation Support Server (Support Server) 11 Network 12 Annotator terminals 15 Input Images 15L Input image for learning 16, 110 local annotation data 20 Training data 30 storage 31 memory 32 CPU 33 Communications Department 34 displays 35 Input Devices 36 Bus Line 40 Operating Programs 41 Region Extraction Models 41LD pre-trained region extraction model 42. Softmax function with temperature 43 Boundary detection filter 50 Read / Write Control Unit (RW Control Unit) 51 Learning Department 60 Encoder section 61 Output section 62 Feature Maps 63 Decoder section 64 Probability Distribution Map Generation Unit 65 Labeling Unit 66 Final Feature Map 67 Probability Distribution Map 67P, 115P processed probability distribution map 68 Output Data 68L Training output data 70 Elements of a probability distribution map Elements of the 70P processed probability distribution map 75, 120, 125, 130, 135, 140 Sharpening treatment 80, 116 Boundary detection process 81, 117 Boundary images 85. Average value calculation process 86 Average 90. Loss calculation processing 91 loss 95. First Total Loss 96, 106 Update settings process 100 Lung lobe size variance calculation process 101. Lung lobe size dispersion 105 Second Total Loss 112 Lung field extraction model 113 Lung field extraction data 126 softargmax function 136A, 136B sigmoid function 200 star-shaped class area 201 yen 202 Star-shaped LB, LB1~LB6 Labels ST100, ST110, ST120, ST130, ST140, ST150, ST160 Step

Claims

1. This is a region extraction model creation support device that assists in creating a region extraction model that extracts regions of multiple classes whose boundaries are contiguous with each other, which are regions of a class that are present in the subject to be recognized in an image. Processor and The processor comprises, The aforementioned processor, The training input image and the local annotation data generated by locally labeling the class region of the training input image are used as training data. The region extraction model outputs a final feature map having element values ​​related to the probability of being in the aforementioned class region. The final feature map, or the probability distribution map that shows the probabilities for each class, generated based on the final feature map, is subjected to sharpening. Based on the results of the sharpening process, the boundary is detected. The boundary length loss, which is the average value of the pixel values ​​in the boundary image generated by detecting the boundary from the result of the sharpening process, is calculated. The region extraction model outputs training output data obtained by extracting the region of the aforementioned class from the training input image. The loss of the region extraction model is calculated according to the comparison result between the local annotation data and the training output data for the local area to which the aforementioned labels have been assigned. The above loss and the boundary length loss are added together to obtain the first total loss. The region extraction model is updated in a direction that reduces the first total loss. A device to support the creation of region extraction models.

2. The image is a medical image, The aforementioned class includes lung lobes, The aforementioned processor, The second overall loss is obtained by further adding the size loss corresponding to the variance or standard deviation of the sizes of multiple lung lobe regions to the first overall loss. The region extraction model creation support device according to claim 1, which updates the region extraction model in a direction that reduces the second overall loss.

3. The region extraction model creation support device according to claim 1, wherein the sharpening process is a process of applying a temperature-enabled softmax function with a temperature parameter of 1 or less to the final feature map or the probability distribution map.

4. The region extraction model creation support device according to claim 1, wherein the sharpening process is a process of applying the softargmax function to the final feature map or the probability distribution map.

5. The region extraction model creation support device according to claim 1, wherein the sharpening process is a process of applying a sigmoid function with a gain of 1 or more to the final feature map or the probability distribution map.

6. The aforementioned processor, The region extraction model creation support device according to claim 1, which accepts the specification of a region to be detected at the boundary in the result of the sharpening process.

7. The region extraction model creation support device according to claim 1, wherein the aforementioned image is a medical image.

8. The region extraction model creation support device according to claim 7, wherein the class includes lung lobes.

9. This is a method for operating a region extraction model creation support device that assists in creating a region extraction model that extracts regions of multiple classes whose boundaries are contiguous, which are regions of a class in an object to be recognized that is captured in an image. The training input image and the local annotation data generated by locally labeling the region of the class in the training input image are used as training data. The region extraction model outputs a final feature map having element values ​​related to the probability of being in the region of the aforementioned class. The final feature map, or the probability distribution map generated based on the final feature map, which shows the probabilities for each class, is subjected to sharpening. The boundary is detected based on the results of the sharpening process. The boundary length loss, which is the average value of the pixel values ​​in the boundary image generated by detecting the boundary from the result of the sharpening process, The region extraction model outputs training output data obtained by extracting the region of the aforementioned class from the training input image. Calculate the loss of the region extraction model according to the comparison result between the local annotation data and the training output data for the local area to which the aforementioned label has been assigned. The first total loss is obtained by adding the aforementioned loss and the boundary length loss, and, The region extraction model is updated in a direction that reduces the aforementioned first total loss. A method for operating a device that supports the creation of region extraction models, including the region extraction model.

10. This is an operating program for a region extraction model creation support device that assists in creating a region extraction model that extracts regions of multiple classes whose boundaries are contiguous, which are regions of a class that are present in an image of a subject to be recognized. The training input image and the local annotation data generated by locally labeling the region of the class in the training input image are used as training data. The region extraction model outputs a final feature map having element values ​​related to the probability of being in the region of the aforementioned class. The final feature map, or the probability distribution map generated based on the final feature map, which shows the probabilities for each class, is subjected to sharpening. The boundary is detected based on the results of the sharpening process. The boundary length loss, which is the average value of the pixel values ​​in the boundary image generated by detecting the boundary from the result of the sharpening process, The region extraction model outputs training output data obtained by extracting the region of the aforementioned class from the training input image. Calculate the loss of the region extraction model according to the comparison result between the local annotation data and the training output data for the local area to which the aforementioned label has been assigned. The first total loss is obtained by adding the aforementioned loss and the boundary length loss, and, The region extraction model is updated in a direction that reduces the aforementioned first total loss. An operating program for a region extraction model creation support device that causes a computer to perform a process including the following.

Images