Image processing device and operation method therefor, inference device, and training device

Abstract

Provided are: an image processing device and an operation method therefor for making it possible to realize higher precision of an output result and speeding up of output in a case of inputting an unknown image; and an inference device and a training device.　A first feature map (41) is extracted by inputting a training input image (21) to a first submodel (40), and a first output image (42) is outputted on the basis of the first feature map (41). A second feature map (51) is extracted by inputting the first feature map (41) to a second submodel (50), and a second output image (52) having higher resolution than the first output image (42) is outputted. By inputting an inference input image to a trained model that has undergone training, the first output image (42) as an inference result image is outputted.

Description

Image processing device and its operating method, inference device, and learning device

[0001] The present invention relates to an image processing device that performs inference on images using machine learning, an operating method thereof, an inference device, and a learning device.

[0002] Patent Document 1 describes a "learning device that provides learning data to a machine learning model for performing semantic segmentation, which is a machine learning model having multiple layers that analyzes an input image, and performs semantic segmentation to distinguish multiple classes contained in the input image on a pixel-by-pixel basis by extracting features that differ in the frequency band of the spatial frequency contained in the input image for each layer, and causes the machine learning model to learn. The learning device includes a reception unit that receives designation of at least one of a necessary band that is estimated to be necessary for learning and an omissible band that is estimated to be omissible in learning from among the multiple frequency bands, and a modification unit that modifies at least one of the machine learning model and the learning data to a form according to the designation received by the reception unit."

[0003] Patent Literature 1 also states that "the decoder network gradually enlarges the image size of the smallest image feature map output from the encoder network. Then, the gradually enlarged image feature map is combined with the image feature maps output at each layer of the encoder network to generate a training output image of the same image size as the training input image." It further states that "the trained model performs semantic segmentation on the input image to determine the class of the object shown in the input image and its contour, and outputs an output image as the result of the determination."

[0004] Japanese Patent Application Laid-Open No. 2020-204863

[0005] In Patent Literature 1, in a machine learning model for performing semantic segmentation, a decoder network performs a process of gradually enlarging image size. When training a machine learning model that performs such segmentation, if the correct answer data is a high-resolution image and high-resolution images are also output when inferring unknown images, the discrimination accuracy of the trained machine learning model when performing inference improves. However, a trained machine learning model that has undergone such training must process high-resolution data, resulting in an increased amount of computation. A decrease in output speed due to the increased amount of computation is undesirable in situations where inference needs to be performed quickly, particularly in near-real-time situations. Therefore, one approach is to reduce the amount of computation by using low-resolution images as the correct answer data. However, using low-resolution correct answer data reduces the amount of information in the data used for training, leading to a deterioration in inference accuracy. Therefore, there is a need for a technology that trains a machine learning model to perform inference on unknown images quickly and with high accuracy.

[0006] An object of the present invention is to provide an image processing device and its operating method, an inference device, and a learning device that realize high-precision output results and high-speed output when an unknown image is input.

[0007] An image processing device of the present invention includes a processor that outputs a first output image based on a first feature map extracted by inputting a training input image to a first sub-model of a training model including a first sub-model and a second sub-model, outputs a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map to the second sub-model, calculates an evaluation result using the second output image, and updates the training model using the evaluation result, thereby making the training model a trained model that includes a first sub-trained model that is the trained first sub-model and a second sub-trained model that is the trained second sub-model, and outputs a first output image as an inference result image based on the first feature map extracted by inputting an inference input image to the first sub-trained model of the trained model.

[0008] The processor calculates the evaluation result by comparing the second output image with a training correct image corresponding to the training input image, and it is preferable that the training correct image is a correct label image in which a correct label is assigned to each area that makes up the training correct image.

[0009] It is preferable that the processor calculates a first evaluation result as an evaluation result by comparing the first output image with a first correct labeled image as a correct labeled image having the resolution of the first output image, and calculates a second evaluation result as an evaluation result by comparing the second output image with a second correct labeled image as a correct labeled image having the resolution of the second output image, and updates the learning model using the first evaluation result and the second evaluation result.

[0010] The first correct labeled image is preferably generated by performing a resolution reduction process on the second correct labeled image.

[0011] The second output image preferably has the same resolution as the training input image, or preferably has a lower resolution than the training input image.

[0012] The first and second sub-models are preferably constructed using convolutional neural networks. The first output image is preferably of lower resolution than the training input image.

[0013] Preferably, the processor further outputs an intermediate feature map having a higher resolution than the first feature map using the first sub-model, and further inputs the intermediate feature map into the second sub-model.

[0014] The learning input images and the inference input images are preferably medical images. The inference input images are preferably images acquired in chronological order.

[0015] It is preferable that the processor generates notification information based on the information contained in the inference result image, generates a notification image based on the notification information, and performs control to display the notification image.

[0016] The notification image is preferably generated so that notification information is superimposed on the inference input image or an image acquired chronologically after the inference input image.

[0017] The notification image is preferably generated so that the inference-use input image or an image acquired chronologically after the inference-use input image and the notification information are displayed at mutually different positions.

[0018] The notification information is preferably position information of a specific shape surrounding an area showing a feature included in the inference input image.

[0019] The method for operating an image processing device of the present invention includes the steps of: outputting a first output image based on a first feature map extracted by inputting a learning input image into a first sub-model, of a learning model including a first sub-model and a second sub-model; outputting a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map into the second sub-model; calculating an evaluation result using the second output image; updating the learning model using the evaluation result to make the learning model a trained model including a first sub-trained model, which is the trained first sub-model, and a second sub-trained model, which is the trained second sub-model; and outputting a first output image as an inference result image based on the first feature map extracted by inputting an inference input image into the first sub-trained model, of the trained models.

[0020] The inference device of the present invention includes a processor. The processor outputs a first output image as an inference result image based on a first feature map extracted by inputting an inference input image into a first sub-trained model among trained models including a first sub-trained model and a second sub-trained model. The trained model is generated by designating the first sub-model as the first sub-trained model and the second sub-model as the second sub-trained model among training models including the first sub-model and the second sub-model. The training model is trained by outputting a first output image based on the first feature map extracted based on the training input image input into the first sub-model, outputting a second output image having a higher resolution than the first output image based on a second feature map extracted based on the first feature map input into the second sub-model, and updating using an evaluation result calculated using the second output image.

[0021] A learning device of the present invention includes a processor. The processor outputs a first output image based on a first feature map extracted by inputting a learning input image into a first sub-model of a learning model including a first sub-model and a second sub-model, outputs a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map into a second sub-model, calculates an evaluation result using the second output image, and updates the learning model using the evaluation result, thereby performing learning. The second output image has a lower resolution than the learning input image.

[0022] According to the present invention, it is possible to achieve high accuracy of the output result and high speed output when an unknown image is input.

[0023] 1 is a schematic diagram of an image processing device. FIG. 1 is a block diagram showing the functions of a learning device. FIG. 2 is a block diagram showing the functions of a learning model. FIG. 3 is an explanatory diagram showing the functions of a first sub-model. FIG. 4 is an explanatory diagram showing the functions of a second sub-model. FIG. 5 is an explanatory diagram showing the functions of an inference device. FIG. 6 is an explanatory diagram showing an example of a training correct answer image in which small regions are classified by assigning three types of class labels. FIG. 7 is an explanatory diagram showing an example of training correct answer image in which small regions are classified by assigning two types of class labels. FIG. 8 is an explanatory diagram showing example of mask data with class labels. FIG. 9 is an explanatory diagram showing the functions of an evaluation unit that calculates multiple evaluation results using multiple training correct answer images with different resolutions. FIG. 10 is an explanatory diagram showing an example of a learning model using Unet. FIG. 11 is an explanatory diagram showing an example of a learning model that performs high resolution processing so that a second output image has a higher resolution than a training input image. FIG. 12 is an explanatory diagram showing an example of a learning model that performs high resolution processing so that a second output image has a lower resolution than a training input image. FIG. 13 is a block diagram showing the functions of a notification control unit. FIG. 14 is an explanatory diagram showing the functions of a notification control unit when position information of a specific shape is generated as notification information. FIG. 15 is an image diagram showing an example of a superimposed image on which position information of a specific shape is superimposed. Fig. 1 is an image diagram showing an example of a notification image that displays position information of a specific shape as a sub-image. Fig. 2 is an explanatory diagram showing the function of a notification control unit when generating position information of a small region as notification information. Fig. 3 is an image diagram showing an example of a superimposed image on which position information of a small region is superimposed. Fig. 4 is an image diagram showing an example of a notification image that displays position information of a small region as a sub-image. Fig. 5 is a flowchart showing an operation method of an image processing device.

[0024] 1, an image processing device 10 includes a learning device 11 and an inference device 12. The learning device 11 and the inference device 12 are connected to each other so as to be able to communicate with each other via a wired connection or wirelessly via a network. The network is, for example, the Internet or a local area network (LAN).

[0025] The image processing device 10 trains the learning model 30 in the learning device 11, and converts the learning model 30 into a trained model 13 that infers the probability of belonging to a small region of an image and extracts a region of interest that is a notable region included in the image. The trained model 13 is transmitted to the inference device 12. An unknown image is input to the inference device 12, and the region of interest included in the unknown image is extracted. A small region of an image refers to a pixel or a set of pixels that make up the image.

[0026] The learning model 30 is a model that performs feature extraction and high-resolution processing on an input image. A control unit (not shown), which is a processor included in the image processing device 10, inputs a learning input image 21 from a learning dataset 20 stored in the data storage unit 14 to the learning model 30. The learning model 30 outputs a first output image 42 from which features of the learning input image 21 have been extracted, and a second output image 52 having a higher resolution than the first output image 42. The learning device 11 updates the learning model 30 using the second output image to create a trained model 13, and transmits the trained trained model 13 to the inference device 12. When an unknown image, an inference input image 121, is input from the modality 15, the trained model 13 performs inference processing on the inference input image 121, which at least extracts features from the image, to output a first output image 42.

[0027] The data storage unit 14 may be provided either externally or internally to the image processing device 10. When the data storage unit 14 is provided externally to the image processing device 10, the training dataset 20 is input from the data storage unit 14 to the learning device 11 via a network. When the data storage unit 14 is provided internally to the image processing device 10, the training dataset 20 is read out by the learning device 11 and input to the learning model 30.

[0028] The specific configuration of the learning device 11 will be described. As shown in FIG. 2 , the learning device 11 includes a learning model 30, an evaluation unit 60, and an update unit 70. The learning model 30 receives a learning input image 21 and outputs a first output image 42 and a second output image 52 using machine learning. The learning model 30 includes a first sub-model 40 that extracts features of the input image and a second sub-model 50 that performs high-resolution processing on the input image data. The learning input image 21 from the learning dataset 20 stored in the data storage unit 14 is input to the first sub-model 40. Note that the number and configuration of the sub-models of the learning model 30 are not limited to those described above, as long as the model as a whole performs feature extraction and high-resolution processing on the input image.

[0029] The first sub-model 40 and the second sub-model 50 are preferably configured using a layered convolutional neural network as shown in Fig. 3. The training input image 21 is input to an input layer 43 of the first sub-model 40. Next, a convolution operation using a plurality of filters is performed at least once in a first intermediate layer 44, which is an intermediate layer of the first sub-model, to extract a first feature map 41 that extracts features of the training input image 21. The first feature map 41 is input to a first output layer 45 and the second sub-model 50.

[0030] The first hidden layer 44 has one or more convolutional layers. In the convolutional layer, a filter is applied to input image data, and a feature map is extracted that indicates the location of the pattern contained in the filter in the input image data. The filter is also called a convolutional kernel. The feature map is also included in the image data input to the convolutional layer. The number of feature maps extracted is equal to the number of filters used in one convolutional layer.

[0031] The first intermediate layer 44 may or may not include a pooling layer. The pooling layer is a layer that summarizes values ​​related to local regions of input image data and performs resolution reduction processing on the image data. The first intermediate layer 44 may be composed of one convolutional layer, but from the viewpoint of improving the accuracy and speed of feature extraction, it is preferable that the first intermediate layer 44 be composed of multiple convolutional layers and pooling layers.

[0032] The first feature map 41 is a feature map output from the latest convolutional layer or pooling layer of the first intermediate layer 44. When the first intermediate layer 44 is composed of multiple convolutional layers and pooling layers, the feature map extracted from the latest layer among the feature maps extracted in the first intermediate layer 44 is defined as the first feature map 41, and the feature map extracted from the layer prior to the first feature map 41 is defined as the first intermediate feature map. A modified example in which the first intermediate layer 44 is composed of multiple layers will be described later.

[0033] The first feature map 41 extracted from the first intermediate layer 44 is input to the first output layer 45. The first output layer 45 uses an activation function to output one first output image 42 from the multiple first feature maps 41. As shown in FIG. 4 , the first output image 42 is classified by calculating the probability of belonging to each region of the input image (the learning input image 21 in FIG. 4 ). For example, the first output image 42 is classified into a region of interest 42 a and a region other than the region of interest 42 b.

[0034] The first feature map 41 extracted from the first hidden layer 44 is further sent to the second hidden layer 54 of the second sub-model 50. The second hidden layer 54 performs at least a process of increasing the resolution of the first feature map 41, and extracts a second feature map 51 (see FIG. 3).

[0035] The second intermediate layer 54 has one or more upsampling layers 54a. The upsampling layers 54a perform enlargement processing (high-resolution processing) of the feature map. The second intermediate layer 54 preferably further has a convolutional layer 54b. While there may be only one upsampling layer 54a and one convolutional layer 54b, from the viewpoint of feature extraction accuracy, it is preferable to have multiple layers.

[0036] Methods for achieving high resolution include, for example, upsampling, which arranges pixel values ​​of pixels constituting a feature map at intervals of several pixels and interpolates the values ​​of the pixels between them, and upconvolution, which combines upsampling without interpolating pixel values ​​with convolution. Upsampling is also called unpooling, and upconvolution is also called transposed convolution or deconvolution. The second intermediate layer 54 may be configured without the upsampling layer 54a. In this case, the second intermediate layer 54 performs the high resolution process using, for example, a shift-and-stitch technique.

[0037] The second feature map 51 is a feature map output from the latest convolutional layer of the second intermediate layer 54. When the second intermediate layer 54 is composed of multiple upsampling layers 54a and convolutional layers 54b, the feature map extracted from the latest layer among the feature maps extracted in the second intermediate layer 54 is defined as the second feature map 51, and the feature map extracted from a layer prior to the second feature map 51 is defined as the second intermediate feature map. In other words, the second feature map 51 is the feature map extracted from the latest layer among the feature maps extracted in the second intermediate layer 54. A modified example in which the second intermediate layer 54 is composed of multiple layers will be described later.

[0038] The second feature maps 51 extracted from the second intermediate layer 54 are input to the second output layer 55. The second output layer 55 uses an activation function similar to the first output layer 45 to output one second output image 52 from the multiple second feature maps 51. Because the second intermediate layer 54 is used to perform a high-resolution process on the first feature map 41, the second output image 52 has a higher resolution than the first output image 42.

[0039] 5, the second output image 52 shows the result of high-resolution processing performed on the first feature map 41, which extracts features (region of interest 41a in FIG. 5) of the input image (learning input image 21 in FIG. 5), and is divided into, for example, a region of interest 52a and a region other than the region of interest 52b. The specific example shown in FIG. 5 shows an example in which the first intermediate layer 44 of the first sub-model 40 performs low-resolution processing on the learning input image 21, and the second intermediate layer 54 of the second sub-model 50 performs high-resolution processing to make the first feature map 41 have a resolution similar to that of the learning input image 21.

[0040] Furthermore, if the second output image 52 has a higher resolution than the first output image 42, it may have a lower resolution than the learning input image 21, the same resolution as the learning input image 21, or a higher resolution than the learning input image 21.

[0041] The second output image 52 is transmitted to the evaluation unit 60 (see FIG. 2 ). The evaluation unit 60 outputs an evaluation result using the second output image 52. For example, in the case of supervised learning, the evaluation unit 60 uses a loss function (also called an error function) that is an evaluation model to output a loss that is the degree of difference between the second output image 52 and the training-use correct image 22, thereby evaluating the accuracy of the output of the learning model 30 as a whole. In this case, the evaluation result 61 is the loss (also called an error) calculated by the evaluation unit 60 using the loss function. The closer the evaluation result 61 is to 0, the smaller the difference between the second output image 52 and the training-use correct image 22, and the higher the output accuracy of the learning model 30.

[0042] The training correct answer image 22 is an image in which the position of a region of interest is indicated in advance, or an image in which one type of class label (correct answer label) from among multiple types of class labels is assigned to each small region, etc. Specific examples of the training correct answer image 22 will be described later.

[0043] The update unit 70 updates the learning model 30 in accordance with the evaluation result calculated by the evaluation unit 60. As a specific example, for example, the update unit 70 updates the network parameters (weights and biases) of the first submodel 40 and the second submodel 50 so that the loss approaches zero. The update unit 70 updates the network parameters so as to minimize the loss, for example, using a stochastic gradient descent method. In this case, the learning rate defines the magnitude of the update amount, and the larger the learning rate, the larger the range of change in the parameters. Note that the update method is not limited to this.

[0044] Semi-supervised learning may be performed using unlabeled learning images in addition to the labeled learning images 22. In this case, the evaluation unit 60 uses a loss function used in supervised learning, and an objective function that satisfies some condition that the unlabeled learning images satisfy, and sets the evaluation result to a calculated value calculated from a function obtained by adding up the loss function and the objective function. The update unit 70 may update the parameters so as to minimize the calculated value calculated from the function obtained by adding up the loss function and the objective function.

[0045] The calculation of the evaluation result 61 by the evaluation unit 60 and the update of the learning model 30 by the update unit 70 are repeated until the evaluation result 61 reaches a preset value. The preset value may be a value within a certain range, or may be greater than or less than a certain threshold value.

[0046] When the evaluation result 61 of the evaluation unit 60 becomes a preset value, the learning model 30 becomes a trained model 13 including a first sub-trained model, which is the trained first sub-model 40, and a second sub-trained model, which is the trained second sub-model 50. The trained model 13 finally generated by the learning device 11 has the same configuration as the learning model 30. For example, if the learning model 30 has the configuration illustrated in FIG. 3, the trained model 13 also has the same configuration.

[0047] The trained model 13 is transmitted from the learning device 11 to the inference device 12 (see FIG. 1). The trained model 13 transmitted from the learning device 11 to the inference device 12 includes a first sub-trained model, which is a trained first sub-model. The trained model 13 transmitted to the inference device 12 may be composed of the first sub-trained model and the second sub-trained model, but is preferably composed of only the first sub-trained model. This is because, from a hardware perspective, there is an advantage in that memory can be saved by omitting the second sub-trained model from the inference device 12.

[0048] As shown in FIG. 6 , the inference device 12 receives an inference input image 121 from the modality 15. The inference input image 121 is input to the input layer 43 of the first sub-trained model of the trained model 13. Next, the first intermediate layer 44 of the first sub-trained model extracts first feature maps 41, and the first output layer 45 outputs one first output image 42 from the multiple first feature maps 41 (see FIG. 3 ). In this example, the first output image 42 output from the first sub-trained model is the inference result image 142. That is, the trained model 13 receives the inference input image 121 and outputs the first output image 42 as the inference result image 142.

[0049] As in this example, by training the learning model 30 so that the second output image 52 has a higher resolution than the first output image 42, the accuracy of the output of the trained model 13 is improved. Furthermore, as in this example, by providing an output layer in the first sub-model (first sub-trained model 13 in the trained model 13), the first output image 42 can be quickly output. In other words, the configuration shown in this example can speed up inference processing for unknown images.

[0050] In machine learning models that perform two different operations, such as feature extraction in one model and high-resolution processing in the other, generally, no output layer is provided between the two models. Therefore, as in this example, a trained model 13 trained on a learning model 30 in which an output layer is provided in the second sub-model that performs high-resolution processing and an output layer is also provided in the first sub-model that performs feature extraction can perform inference processing faster and with higher recognition accuracy than general machine learning models. In other words, the trained model 13 in this example can achieve highly accurate output in almost real time for an unknown image input.

[0051] If the trained model 13 is composed of a first sub-trained model and a second sub-trained model, when outputting the inference result image 142, the second output image may be output from the second sub-trained model, but the second output image is not used to generate alarm information. When the inference input image 121 is input to the trained model 13, it is preferable to use only the first sub-trained model, not the second sub-trained model, and not output the second output image. When the inference input image 121, which is an unknown image, is input to the trained model 13, the rapid output of the first output image 42 can be sufficiently achieved by installing the first sub-trained model in the inference device 12, but by outputting the inference result image 142 using only the first sub-trained model, the calculation processing within the inference device 12 can be further accelerated.

[0052] Furthermore, if the second sub-trained model is not used when outputting the inference result image 142, it is preferable not to input the first feature map extracted by the first sub-trained model to the second sub-trained model.

[0053] The evaluation unit 60 preferably compares the second output image 52 with the training correct image 22, and calculates an evaluation result 61 that evaluates the calculation of the probability of belonging for each small region and the accuracy of classification. The training correct image 22 used in the learning device 11 is preferably a correct-label image in which a correct label is assigned to each region that makes up the training correct image 22. The correct label refers to a class label that indicates "correct" and is assigned to each small region that makes up the training correct image 22.

[0054] For example, in the specific example of Figure 7, the small area 22a constituting the correct learning image 22 is assigned a correct label 23a of "normal mucosa," the small area 22b is assigned a correct label 23b of "inflammation," and the small area 22c is assigned a correct label 23c of "malignant tumor."

[0055] 8, the correct learning image 22 may be divided into an attention region and a region other than the attention region, and correct labels may be assigned to the regions. In the specific example of FIG. 8, a small region 22d constituting the correct learning image 22 is assigned a correct label 23d of "normal region" as a region other than the attention region, and a small region 22e is assigned a correct label 23e of "abnormal region" as a region of interest. Examples of correct labels are not limited to this.

[0056] The specific examples of Figures 7 and 8 show a training correct answer image 22 in which correct answer labels are assigned to small regions corresponding to a training input image 21 in which structures such as mucosal folds and redness due to inflammation can be visually distinguished. On the other hand, as shown in Figure 9, the training correct answer image 22 is preferably mask data in which each small region to which a correct answer label is assigned is divided into different colors, so that structures such as mucosal folds and redness due to inflammation cannot be visually distinguished. The specific example of Figure 9 shows a training correct answer image 22 in which correct answer labels 23a, 23b, and 23c are assigned to each of small regions 22a, 22b, and 22c, as in Figure 7, and it is possible to determine only which class each small region belongs to.

[0057] 7 to 9 are used, the learning model 30 is a model that performs segmentation, and the first output image 42 and the second output image 52 each have a predicted class label for each small region that makes up the learning input image 21. With the above configuration, the trained model 13 can be a model that performs segmentation on an unknown image and detects a region of interest with high accuracy and high speed.

[0058] An area of ​​interest is an area to which a user should pay attention. For example, in the case of a medical image, the area of ​​interest refers to an area of ​​the medical image that shows an abnormality such as a malignant tumor, a benign tumor, a polyp, inflammation, bleeding, vascular irregularity, glandular irregularity, hyperplasia, dysplasia, trauma, or fracture, or an abnormal area in a living body, such as a scar, a surgical scar, a medical solution, a fluorescent dye, an artificial joint, an artificial bone, or a foreign body such as gauze, or an area where a procedure has been performed on the living body. Furthermore, in the case of an image of a product of a machine tool as the subject, an area of ​​interest is, for example, an area that shows an abnormality in the product, such as a crack, tear, or scratch. However, examples of an area of ​​interest are not limited to these.

[0059] Alternatively, the training correct image 22 may be an image in which a correct label is attached only to the region of interest. In this case, the training model 30 may output a class label only to the small region that is the region of interest, without outputting a class label to small regions other than the region of interest.

[0060] The classification of small regions and the assignment of class labels to the training answer image 22 may be performed in advance by a user or may be performed using machine learning installed in a device other than the image processing device 10. The user may be, for example, a doctor who is skilled in diagnosing medical images.

[0061] It is preferable that the evaluation result be calculated by comparing the training correct answer image 22 with the first output image 42 in addition to comparing the training correct answer image 22 with the second output image 52. That is, while Fig. 2 shows a specific example in which the evaluation result 61 is calculated by comparing the training correct answer image 22 with the second output image 52, it is also preferable that the evaluation result be calculated by comparing the training correct answer image 22 with the first output image 42.

[0062] In this case, the training dataset 20 includes two types of training correct answer images 22 (second correct answer labeled images) having different resolutions: a training correct answer image 22 (first correct answer labeled image) having the resolution of the first output image 42, and a training correct answer image 22 having the resolution of the second output image 52. Note that the closer the resolutions of the first correct answer labeled image and the first output image 42, the more preferable, and they are the same. Similarly, the closer the resolutions of the second correct answer labeled image and the second output image 52, the more preferable, and they are the same. The resolutions of the first correct answer labeled image and the second correct answer labeled image are different from each other, and the resolution of the second correct answer labeled image is higher than the resolution of the first correct answer labeled image.

[0063] 10 , the evaluation unit 60 compares the first output image 42 output by the first sub-model 40 by inputting the learning input image 21 into the first sub-model with the first supervised labeled image 24, and calculates a first evaluation result 62 as the evaluation result. Furthermore, the evaluation unit 60 compares the second output image 52 output by the second sub-model 50 with the second supervised labeled image 25, and calculates a second evaluation result 63 as the evaluation result.

[0064] The calculated first evaluation result 62 and second evaluation result 63 are input to the update unit 70. The update unit 70 updates the learning model 30 based on the first evaluation result 62 and the second evaluation result 63. The first evaluation result 62 is a loss indicating the difference between the first output image 42 and the first correct-labeled image 24, and the second evaluation result 63 is a loss indicating the difference between the second output image 52 and the second correct-labeled image 25. With the above configuration, the learning model 30 can be updated using two types of evaluation results, thereby further improving the accuracy of learning.

[0065] The first correct label image 24 and the second correct label image 25 may be generated separately, but it is preferable that the first correct label image 24 be generated by performing a resolution reduction process on the second correct label image 25. In this case, the image processing device 10 may be provided with a first correct label image generation unit (not shown), and the first correct label image generation unit may reduce the resolution of the second correct label image 25 to generate the first correct label image 24, or a device other than the image processing device 10 may reduce the resolution of the second correct label image 25 to generate the first correct label image 24. With the above configuration, the second correct label image 25 can be generated at low cost without newly generating the first correct label image 24.

[0066] If the second output image 52 output by the second sub-model has a higher resolution than the first output image 42 output by the first sub-model, the first sub-model 40 may output the first output image 42 by performing an operation to reduce the resolution of the learning input image 21, or may output the first output image 42 with the same resolution as the learning input image 21. Furthermore, the second sub-model 50 may output the second output image 52 with the same resolution as the learning input image 21, may output the second output image 52 with a higher resolution than the learning input image 21, or may output the second output image 52 with a lower resolution than the learning input image 21.

[0067] The combination of processes performed in the first sub-model 40 and the second sub-model 50 will be described.

[0068] (1) A learning model 30 in which feature extraction and low-resolution processing are performed in the first sub-model 40, and high-resolution processing is performed in the second sub-model 50 so that the second output image 52 has the same resolution as the learning input image 21.

[0069] (2) A learning model 30 in which feature extraction and resolution reduction processing are performed in the first sub-model 40, and resolution increase processing is performed in the second sub-model 50 so that the second output image 52 has a higher resolution than the learning input image 21.

[0070] (3) A learning model 30 in which feature extraction and low-resolution processing are performed in the first sub-model 40, and high-resolution processing is performed in the second sub-model 50 so that the second output image 52 has a lower resolution than the learning input image 21 (however, the second output image 52 has a higher resolution than the first output image 42).

[0071] (4) A learning model 30 in which the first sub-model 40 does not undergo a resolution reduction process, and the second sub-model 50 undergoes a resolution increase process so that the second output image 52 has a higher resolution than the learning input image 21.

[0072] It is preferable that the first output image 42 has a lower resolution than the training input image 21. The output speed of the first output image 42 of the trained model 13 that is finally generated is faster when the first output image 42 has a lower resolution than the training input image 21 than when the first output image 42 has the same resolution as the training input image 21. In other words, by having the first sub-model 40 perform resolution reduction processing, the inference processing speed of the trained trained model 13 can be improved. In the above-mentioned examples of the training models 30 (1) to (4), the learning models 30 (1) to (3) in which the first sub-model performs resolution reduction processing output the first output image 42 faster than the learning model 30 (4).

[0073] Furthermore, by performing a resolution reduction process on the first sub-model 40, it is possible to extract a first feature map 41 that aggregates information from a wider range of the image. For example, when performing a convolution process on a high-resolution image and an edge is extracted from that image, it may be difficult to accurately recognize and classify a small region containing the extracted edge as normal mucosa or a polyp. To address this issue, the feature map obtained by convolution is reduced in resolution to further aggregate information, and by repeating the convolution process again, a wider range of information is aggregated, which may enable the edge to be determined to be a polyp.

[0074] The first sub-model 40 extracts the first feature map 41, which aggregates information over a wide range through a resolution reduction process, and the second sub-model 50 increases the resolution of the aggregated first feature map 41. This restores the position information of the aggregated local feature information across the entire image, and updates the learning model 30 so that the extracted features and their position information become accurate. The trained model 13 that has undergone such training can perform highly accurate recognition even for unknown, high-resolution images. In particular, in segmentation, which classifies images into small regions, recognition accuracy can be improved by training to accurately determine the position information of features.

[0075] The higher the resolution of the second feature map 51 and the second output image 52 based on the second feature map, the more learning can be performed to improve the output accuracy of the learning model 30. Accordingly, the accuracy of the inference process of the trained model 13 improves. In the examples of the learning models 30 (1) to (4) described above, the learning models 30 (2) and (4), which perform high-resolution processing in the second sub-model 50 to make the second output image 52 higher in resolution than the learning input image 21, have higher output accuracy for the learning input image 21 than the learning models 30 (1) and (3).

[0076] On the other hand, in learning a learning model using segmentation, the higher the resolution of the final output image, the more likely overfitting occurs due to the increase in parameters used in learning. Therefore, by outputting the second output image 52 at a lower resolution than the learning input image 21, learning can be stabilized and overfitting can be suppressed. In this way, when the second output image 52 has a higher resolution than the learning input image 21, there is a trade-off between improving the accuracy of inference for the learning input image 21 and overfitting, which reduces the recognition accuracy for unknown images. Of the examples of the learning model 30 (1) to (4) described above, by providing the learning device 11 with the learning model 30 (3), in which the second sub-model 50 performs a high-resolution process to lower the resolution of the second output image 52 than the learning input image 21, the learning device 11 can be made to be able to suppress overfitting.

[0077] Furthermore, in addition to the first feature map 41 extracted from the first sub-model 40, it is preferable to input an intermediate feature map (first intermediate feature map) to the second sub-model 50. Known examples of the learning model 30 having such a configuration include ResNet (Residual Network) and Unet (U-shaped Network).

[0078] The use of Unet in the learning model 30 will be described using the specific example shown in Fig. 11. The first hidden layer 44 (see Fig. 3) of the first sub-model 40 has multiple convolutional layers 44a, 44c, 44e, and 44g, and multiple pooling layers 44b, 44d, and 44f.

[0079] The pooling layer 44b downsamples the feature map input from the convolutional layer 44a to reduce the resolution of the feature map. Similarly, the pooling layer 44d reduces the resolution of the feature map input from the convolutional layer 44c, and the pooling layer 44f reduces the resolution of the feature map input from the convolutional layer 44e. The pooling layers 44b, 44d, and 44f provide robustness to the position information of the extracted features and further contribute to extracting features necessary for class classification.

[0080] 11 , the feature map extracted from the most recent convolutional layer 44 g is the first feature map 41. The feature maps extracted from the convolutional layers 44 a, 44 c, and 44 e other than the convolutional layer 44 g are first intermediate feature maps.

[0081] The second hidden layer 54 (see FIG. 3 ) of the second sub-model 50 includes multiple upsampling layers 54c, 54e, and 54g and multiple convolutional layers 54d, 54f, and 54h. The upsampling layer 54c increases the resolution of the first feature map 41 input from the convolutional layer 44g of the first sub-model 40. Similarly, the upsampling layer 54e increases the resolution of the feature map input from the convolutional layer 54d, and the upsampling layer 54g increases the resolution of the feature map input from the convolutional layer 54f.

[0082] 11 , the feature map extracted from the convolutional layer 54h, which is the most recent layer, is the second feature map 51. The feature maps extracted from the convolutional layers 54d and 54f other than the convolutional layer 54h and the upsampling layers 54c, 54e, and 54g are second intermediate feature maps.

[0083] In Unet, layers that perform convolution of intermediate feature maps of similar resolution are paired, and the intermediate feature map (first intermediate feature map 41b) extracted by the sub-model that performs downsampling is input to the paired layer of the sub-model that performs upsampling. In the specific example of FIG. 11, the paired layers are as follows: (1; first layer) A layer consisting of the convolution layer 44a and the pooling layer 44b, and a layer consisting of the upsampling layer 54g and the convolution layer 54h. (2; second layer) A layer consisting of the convolution layer 44c and the pooling layer 44d, and a layer consisting of the upsampling layer 54e and the convolution layer 54f. (3; third layer) A layer consisting of the convolution layer 44e and the pooling layer 44f, and a layer consisting of the upsampling layer 54c and the convolution layer 54d. In the first sub-model 40, the resolution is reduced stepwise from the first layer to the third layer, and in the second sub-model 50, the resolution is increased stepwise from the third layer to the first layer.

[0084] 11 , in the first layer, the first intermediate feature map 41b extracted by the convolutional layer 44a is input to the convolutional layer 54h. In the second layer, the first intermediate feature map 41b extracted by the pooling layer 44d is input to the convolutional layer 54f. In the third layer, the first intermediate feature map 41b extracted by the pooling layer 44f is input to the convolutional layer 54d.

[0085] In this way, by inputting the first intermediate feature map 41b extracted by the first sub-model 40 into the second sub-model 50, it becomes easier to recover the spatial resolution once lost in the downsampling process, which is considered difficult, and high-precision learning can be performed. Furthermore, the recovery of the spatial resolution is performed by combining the first intermediate feature map 41b and the second intermediate feature map, for example, by adding them together.

[0086] Note that, as in Unet, intermediate feature maps may be passed between paired layers, or the first intermediate feature map extracted by the first submodel 40 may be increased in resolution and the increased-resolution first intermediate feature map may be input to the second submodel 50. That is, in Unet, intermediate feature maps may be passed to layers other than the paired layer. This method also makes it easier to restore spatial resolution when upsampling.

[0087] 12 shows that the number of upsampling layers 54c, 54e, and 54g in the second sub-model 50 is greater than the number of pooling layers 44b and 44d in the first sub-model 40, thereby performing a resolution enhancement process so that the second output image 52 has a higher resolution than the training input image 21. That is, this shows an example of a learning model 30 in which (2) above, feature extraction and resolution reduction process are performed in the first sub-model 40, and resolution enhancement process is performed in the second sub-model 50 so that the second output image 52 has a higher resolution than the training input image 21. In this case, the first intermediate feature map extracted from the convolution layer 44a of the first sub-model 40 may be increased in resolution and input to the convolution layer 54h of the second sub-model 50.

[0088] 13 shows that the number of upsampling layers 54c, 54e of the second sub-model 50 is made smaller than the number of pooling layers 44b, 44d, 44f of the first sub-model 40, thereby performing high-resolution processing so that the second output image 52 has a lower resolution than the training input image 21. That is, this shows an example of a learning model 30 in which (3) above, feature extraction and low-resolution processing are performed in the first sub-model 40, and high-resolution processing is performed in the second sub-model 50 so that the second output image 52 has a lower resolution than the training input image 21 (however, the second output image 52 has a higher resolution than the first output image 42).

[0089] Although the above discloses an example in which the learning model 30 has two sub-models, the learning model 30 may have one machine learning model as long as it includes an input layer 43, a first intermediate layer 44 that extracts the first feature map 41 by performing feature extraction, a first output layer 45 that outputs the first output image 42 based on the first feature map 41, a second intermediate layer 54 that receives the first feature map 41 as input and extracts the second feature map 51 by performing high-resolution processing on at least the first feature map 41, and a second output layer 55 that outputs the second output image 52 based on the second feature map 51. In other words, the learning model 30 disclosed in this embodiment can be obtained by configuring a machine learning model such that an intermediate layer and an output layer that perform feature extraction are provided before the intermediate layer that performs high-resolution processing, and another output layer is provided after the intermediate layer that performs high-resolution processing.

[0090] The learning input images 21 and the inference input images 121 are preferably medical images. Medical images are images acquired by a modality 15 such as an endoscope, a radiography device, an ultrasound imaging device, or a nuclear magnetic resonance device, and are used by doctors and other medical professionals for diagnosis. Specific examples of medical images include endoscopic images, radiological images such as X-ray images, computed tomography (CT) images, ultrasound images, and magnetic resonance imaging (MRI) images.

[0091] The learning model 30, which has been trained using a medical image as the learning input image 21, is used as the trained model 13, and further, by performing inference using the trained model 13 with the medical image as the inference input image 121, it is possible to recognize regions of interest on medical images with high accuracy and speed, and to support the diagnosis made by a doctor user, thereby improving the accuracy of diagnosis. Furthermore, the learning device 11 of this example can perform training to achieve high output accuracy even in the medical field, where the amount of image data used as the learning dataset 20 generally tends to be small.

[0092] The learning input images 21 and the inference input images 121 may be images other than medical images. For example, they may be images including roads, cars, and people as subjects, which are acquired using a drive recorder as the modality 15.

[0093] The inference input images 121 are preferably images acquired in chronological order. For example, when a flexible endoscope inserted into the digestive tract of a patient is used as the modality 15, the inference input images 121 are endoscopic images of the mucosal surface of the digestive tract acquired in chronological order as the doctor moves the tip of the endoscope from the rectum to the ileocecal region.

[0094] Furthermore, when an ultrasound imaging diagnostic device that emits ultrasound by contacting a probe with the skin of a patient's abdomen is used as the modality 15, the inference input image 121 is an ultrasound image. An ultrasound image is a medical image acquired while undergoing time-series changes in accordance with the patient's breathing and heartbeat.

[0095] The inference result image 142 output by the trained model 13 of the inference device 12 is transmitted to the notification control unit 80 of the image processing device 10 (see FIG. 6 ). As shown in FIG. 14 , the notification control unit 80 includes a notification information generation unit 90 and a notification image generation unit 100.

[0096] The notification information generation unit 90 generates notification information based on information extracted from the inference result image 142 about features of the inference input image 121. The notification information is information indicating the position of the inference input image 121 containing the attention area, which is a feature extracted in the trained model 13. The notification image generation unit 100 uses the notification information to generate a notification image, which is an image that displays the notification information.

[0097] The notification image is preferably a superimposed image in which notification information is superimposed on an image acquired by the modality 15. There is also a sub-image, which is an image in which notification information is displayed at a position different from the position at which the image acquired by the modality 15 is displayed.

[0098] The image acquired by the modality 15 is preferably the inference input image 121 or an image acquired chronologically after the inference input image 121. When the inference result image 142 is output almost simultaneously with the acquisition of the inference input image 121, the position of the attention area indicated by the notification information remains almost unchanged even for an image acquired chronologically after the inference input image 121 (particularly, immediately after, for example, several frames later). Therefore, even if a notification image (superimposed image or sub-image) is generated using an image acquired chronologically after the inference input image 121 and the notification information, the user can recognize the position of the attention area included in the notification information.

[0099] The notification information is preferably position information of a specific shape surrounding a region showing a feature included in the inference input image 121 transmitted from the modality 15. The specific shape is, for example, a bounding box surrounding the region of interest. Note that the shape of the specific shape is not limited to a rectangle, but may also be an ellipse or a polygon. Furthermore, the display mode, such as color, of the specific shape may be set arbitrarily or automatically. Furthermore, when the trained model 13 performs segmentation and detects multiple regions of interest as features, and the regions of interest are classified into multiple classes, such as "polyp" and "inflammation," the display mode, such as shape and color, of the specific shape may be different for each class. In addition, class labels, such as "polyp" and "inflammation," may be displayed near the specific shape.

[0100] The flow of generating a notification image when the notification information is position information of a specific shape surrounding an area indicating a feature included in the inference input image 121, and a specific example of the generated notification image will be described below. First, a case where the notification image is a superimposed image will be illustrated using FIG. 15 . The inference input image 121 is input to the trained model 13, and an inference result image 142 is output as the first output image 42. The inference result image 142 includes an area of ​​interest 142a as the extracted feature 121a. In the specific example shown in FIG. 15 , the output inference result image 142 has a lower resolution than the inference input image 121, as indicated by the smaller size of the inference result image 142. Furthermore, the feature 121a of the inference input image 121 that has been subjected to resolution reduction processing is shown to be classified as the area of ​​interest 142a.

[0101] Next, the notification information generation unit 90 generates notification information 91 from the inference result image 142. In the specific example shown in Fig. 15, the notification information 91 is position information of a rectangle 91a that surrounds the extracted attention area 142a. Note that in Fig. 15, the attention area 142a is shown by a dashed line for the sake of explanation, but the notification information generation unit 90 generates only the position information of the rectangle 91a as the notification information 91.

[0102] The generated notification information 91 is transmitted to the notification image generation unit 100. Furthermore, an image from the modality 15 (the inference input image 121 or an image acquired chronologically after the inference input image 121) is transmitted to the notification image generation unit 100. The notification image generation unit 100 superimposes the notification information 91 on the image from the modality 15 to generate a superimposed image 101 as shown in FIG. 16. The superimposed image 101 has position information of a rectangle 91a superimposed thereon as the notification information 91. The superimposed image 101 is transmitted to the display control unit 110 (see FIG. 6).

[0103] The display control unit 53 controls the display of the notification image generated by the notification image generation unit 100 on the display 16 (see FIG. 6). Finally, the notification image that can be visually recognized by the user is displayed on the display 16.

[0104] As in the above example, by displaying the notification information 91 on the display 16 as the superimposed image 101, the user can recognize the notification information without moving his / her line of sight.

[0105] Next, a modified example of the notification image in which notification information 91, which is the position information of a rectangle 91a, is displayed as a sub-image is described. The process of transmitting the notification information 91 and the image from the modality 15 to the notification image generation unit 100 is the same as the example described with reference to FIG. 15 . In this case, as shown in FIG. 17 , the notification image 103 generated by the notification image generation unit 100 has a main section 103a displaying the image 15a from the modality 15 and a sub-section 103b displaying a sub-image 104, which is an image displaying the notification information 91 (a rectangle 91a indicating the position information of the region of interest 142a). The main section 103a and the sub-section 103b may have any positional relationship as long as they are located at different positions on the notification image 103. Furthermore, the sizes of the main section 103a and the sub-section 103b can be set arbitrarily. The notification image 103 is transmitted to the display control unit 110.

[0106] Depending on the situation, it may not be desirable to superimpose notification information on an image from modality 15 displayed on display 16. For example, if the user is a physician, he or she may want to closely observe an image including a region of interest, such as a lesion. In such a situation, superimposing notification information on the image may actually interfere with the user's observation. For this reason, by displaying notification information 91 as a sub-image, as in the above-described modified example, it is possible to display the position information of the region of interest to be observed without interfering with the user's observation.

[0107] Next, a modified example in which position information of a small region classified as an attention region from the inference input image 121 is generated as notification information, and an notification image showing the position information of the small region in a specific color is generated will be described using the specific example shown in Fig. 18. First, an example in which a superimposed image is generated as the notification image will be described. In this case, as in the example shown in Fig. 15, the inference input image 121 is input to the trained model 13, and an inference result image 142 including an attention region 142a as the extracted feature 121a is output and transmitted to the notification information generation unit 90.

[0108] As shown in FIG. 18 , the notification information generation unit 90 generates, as notification information 92, position information of a small region 92a, which is the extracted region of interest 142a. As shown in FIG. 19 , the notification image generation unit 100 superimposes an image, in which the position information of the small region 92a, as notification information 92, is represented in a specific color on an image from the modality 15, to generate a superimposed image 101. The position information of the small region 92a, represented in a specific color, is superimposed on the superimposed image 101 as notification information 92. The position information of the small region 92a, represented in a specific color, is preferably superimposed with its transparency adjusted so that the image from the modality 15, which is the background, can be seen through. The superimposed image 101 is transmitted to the display control unit 110. Note that the specific color can preferably be arbitrarily set to match the modality 15. With the above configuration, the user can recognize the region of interest as a color distribution.

[0109] Furthermore, a modified example will be described in which the notification image displays notification information 92, which is position information of a small region 92a indicated in a specific color, as a sub-image. The flow until the notification information 92 and the image from the modality 15 are transmitted to the notification image generation unit 100 is the same as the example described using FIG. 18 . In this case, as shown in FIG. 20 , the notification image 103 displays the image 15a from the modality 15 in the main section 103a and the notification information 92 as a sub-image 104 in the sub-section 103b. It is preferable that the sub-image 104 is a minimap that displays position information of the small region 92a in a specific color. With the above configuration, the distribution of the attention area can be visualized and recognized by the user without interfering with the user's observation.

[0110] A series of steps in the operation method of the image processing device 10 of this embodiment will be described with reference to the flowchart in Fig. 21. First, a learning input image 21 is input to a first sub-model 40 of the learning model 30 (step ST101). A first feature map 41 is extracted from the learning input image 21 using the first sub-model 40 (step ST102), and a first output image 42 is output based on the first feature map 41 (step ST103). Next, the first feature map 41 is input to a second sub-model 50 (step ST104). A second feature map 51 is extracted from the first feature map 41 using the second sub-model 50 (step ST105), and a second output image 52 with a higher resolution than the first output image 42 is output based on the second feature map 51 (step ST106).

[0111] Next, the evaluation unit 60 calculates the evaluation result 61 using the second output image 52 (step ST107). The update unit 70 updates the parameters of the learning model 30 using the evaluation result 61 (step ST108). Through repeated updates, the learning model 30 is generated as a trained model 13 (step ST109). Finally, by inputting the inference input image 121 to the trained model 13 that has completed learning (step ST110), an inference process of the trained model 13 is performed, and the first output image 42 is output from the trained model 13 as an inference result image 142 (step ST111).

[0112] In this embodiment, the term "image" refers to image data, including the training input image 21, the training correct image 22, the inference input image 121, the inference result image 142, the first output image 42, the second output image 52, the first feature map 41, the second feature map 51, the first intermediate feature map, the second intermediate feature map, the correct-label image, the first correct-label image 24, the second correct-label image 25, the image from the modality 15, the notification images 101 and 103, and the sub-image 104.

[0113] In the image processing device 10, programs related to various processes or controls are stored in a program storage memory (not shown). A control unit (not shown) configured by a processor executes the programs stored in the program storage memory to realize the functions of the learning device 11, the inference device 12, the notification control unit 80, and the display control unit 110. Note that the learning device 11 may be separated from the image processing device 10. In this case, the learning device 11 may be provided with a first control unit configured by a processor, and the image processing device 10 may be provided with a second control unit configured by a processor.

[0114] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the learning device 11, the inference device 12, the notification control unit 80, the display control unit 110, and the control unit, is the following various processors: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, which is a processor with a circuit configuration designed specifically for executing various processes.

[0115] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs or a combination of a CPU and an FPGA). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0116] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0117] 10 Image processing device 11 Learning device 12 Inference device 13 Trained model 14 Data storage unit 15 Modality 15a Image from modality 16 Display 20 Training dataset 21 Training input image 22 Training correct image 22a, 22b, 22c, 22d, 22e, 92a Small region 23a, 23b, 23c, 23d, 23e Correct label 24 First correct labeled image 25 Second correct labeled image 30 Training model 40 First sub-model 41 First feature map 41a, 42a, 52a, 142a Attention region 41b First intermediate feature map 42 First output image 42b, 52b Region other than attention region 43 Input layer 44 First intermediate layer 44a, 44c, 44e, 44g, 54b, 54d, 54f, 54h Convolution layer 44b, 44d, 44f Pooling layer 45 First output layer 50 Second sub-model 51 Second feature map 52 Second output image 55 Second hidden layer 54a, 54c, 54e, 54g Upsampling layer 55 Second output layer 60 Evaluation unit 61 Evaluation result 62 First evaluation result 63 Second evaluation result 70 Update unit 80 Notification control unit 90 Notification information generation unit 91, 92 Notification information 91a Rectangle 100 Notification image generation unit 101 Superimposed image 103 Notification image 103a Main section 103b Sub section 104 Sub-image 110 Display control unit 121 Input image for inference 121a Feature 142 Inference result image

Claims

1. a processor; The processor: outputting a first output image based on a first feature map extracted by inputting a learning input image into the first sub-model of a learning model including a first sub-model and a second sub-model; outputting a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map into the second sub-model; Calculating an evaluation result using the second output image; By updating the learning model using the evaluation result, the learning model becomes a trained model including a first sub-trained model that is the trained first sub-model and a second sub-trained model that is the trained second sub-model; An image processing device that outputs the first output image as an inference result image based on the first feature map extracted by inputting an input image for inference to the first sub-trained model among the trained models.

2. The processor: calculating the evaluation result by comparing the second output image with a learning answer image corresponding to the learning input image; The image processing device according to claim 1 , wherein the training correct answer image is a correct answer labeled image in which a correct answer label is attached to each of areas constituting the training correct answer image.

3. The processor: calculating a first evaluation result as the evaluation result by comparing the first output image with a first correct-labeled image as the correct-labeled image having a resolution of the first output image, and calculating a second evaluation result as the evaluation result by comparing the second output image with a second correct-labeled image as the correct-labeled image having a resolution of the second output image; The image processing device according to claim 2 , wherein the learning model is updated using the first evaluation result and the second evaluation result.

4. The image processing device according to claim 3 , wherein the first correct labeled image is generated by performing a resolution reduction process on the second correct labeled image.

5. The image processing device according to claim 1 , wherein the second output image has the same resolution as the learning input image.

6. The image processing device according to claim 1 , wherein the second output image has a lower resolution than the learning input image.

7. The image processing device according to claim 1 , wherein the first sub-model and the second sub-model are configured using a convolutional neural network.

8. The image processing device according to claim 1 , wherein the first output image has a lower resolution than the learning input image.

9. The processor: further outputting an intermediate feature map having a higher resolution than the first feature map using the first sub-model; The image processing apparatus according to claim 1 , further comprising: inputting the intermediate feature map into the second sub-model.

10. The image processing apparatus according to claim 1 , wherein the learning input images and the inference input images are medical images.

11. The image processing apparatus according to claim 1 , wherein the input images for inference are images acquired in chronological order.

12. The processor: generating notification information based on information contained in the inference result image; generating a notification image based on the notification information; The image processing device according to claim 1 , wherein the image processing device controls display of the notification image.

13. The image processing device according to claim 12 , wherein the notification image is generated so as to superimpose the notification information on the input image for inference or an image acquired chronologically after the input image for inference.

14. The image processing device according to claim 12, wherein the notification image is generated so as to display the input image for inference or an image acquired chronologically after the input image for inference and the notification information at mutually different positions.

15. The image processing device according to claim 13 , wherein the notification information is position information of a specific shape surrounding an area showing a feature included in the inference input image.

16. outputting a first output image based on a first feature map extracted by inputting a learning input image to a first sub-model of a learning model including a first sub-model and a second sub-model; outputting a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map into the second sub-model; calculating an evaluation result using the second output image; updating the learning model using the evaluation result to convert the learning model into a trained model including a first sub-trained model that is the trained first sub-model and a second sub-trained model that is the trained second sub-model; A method for operating an image processing device, comprising a step of outputting the first output image as an inference result image based on the first feature map extracted by inputting an inference input image into the first sub-trained model among the trained models.

17. 1. An inference device comprising a processor, The processor: outputting a first output image as an inference result image based on a first feature map extracted by inputting the inference input image into a first sub-trained model among trained models including a first sub-trained model and a second sub-trained model; The trained model is generated by setting the first sub-model as the first sub-trained model and the second sub-model as the second sub-trained model, of a training model including a first sub-model and a second sub-model; The learning model is an inference device that learns by outputting a first output image based on the first feature map extracted based on a learning input image input to the first sub-model, and outputting a second output image having a higher resolution than the first output image based on a second feature map extracted based on the first feature map input to the second sub-model, and updating the second output image using an evaluation result calculated using the second output image.

18. A learning device comprising a processor, The processor: outputting a first output image based on a first feature map extracted by inputting a learning input image into the first sub-model of a learning model including a first sub-model and a second sub-model; outputting a second output image having a higher resolution than the first output image based on a second feature map extracted by inputting the first feature map into the second sub-model; Calculating an evaluation result using the second output image; performing learning by updating the learning model using the evaluation results; A learning device in which the second output image has a lower resolution than the learning input image.