Inference model learning system, inference device, inference model learning method, inference method, and program

Abstract

Provided is an inference model learning system comprising: a first acquisition unit that obtains a first data set and a second data set; a second acquisition unit that obtains an intermediate learning data set from at least one of the first data set and the second data set, on the basis of a first feature amount included in a first feature amount group corresponding to the first data set and a second feature amount included in a second feature amount group corresponding to the second data set; and a learning unit that performs first learning processing on an inference model using a data set included in the first data set, performs second learning processing on a learned model obtained by the first learning processing using the intermediate learning data set, and performs third learning processing on a learned model obtained by the second learning processing using a data set included in the second data set.

Description

Inference Model Learning System, Inference Device, Inference Model Learning Method, Inference Method, and Program

[0001] The present disclosure relates to an inference model learning system, an inference device, an inference model learning method, an inference method, and a program.

[0002] In the field of machine learning, when learning a machine learning model with a target predetermined dataset, there is a technique called transfer learning that efficiently performs learning by using a learned model that has been learned with another large-scale dataset. When performing transfer learning, it is known that the learning effect is higher when the difference in feature amounts between the two datasets used for learning is small.

[0003] In the technique described in Non-Patent Document 1, after using ImageNet, which is a large-scale database, for pre-training, before performing additional learning with a tomosynthesis image group, learning of a pre-trained model (hereinafter referred to as intermediate learning) is performed with a mammography image group. Thereby, the difference in feature amounts between the datasets for performing transfer learning is reduced. Also, in the technique described in Patent Document 1, by extracting a dataset with feature amounts close to those of the second dataset from the first dataset, the difference in feature amounts between the two datasets for performing transfer learning is reduced.

[0004] Japanese Unexamined Patent Application Publication No. 2016-191966

[0005] Ravi K. Samala et al., Breast Cancer Diagnosis in Digital Breast Tomosynthesis: Effects of Training Sample Size on Multi-Stage Transfer Learning using Deep Neural Nets, IEEE Trans Med Imaging. 2019, P.686-696

[0006] However, in the technique described in Non-Patent Document 1, it is necessary to externally acquire a dataset for intermediate learning separately from the first dataset and the second dataset, and transfer learning could not be efficiently performed.

[0007] One embodiment of this disclosure aims to provide a learning system that can efficiently perform transfer learning even when there is a large difference in the features depicted in the two datasets when performing transfer learning using a pre-training dataset and an additional training dataset.

[0008] A learning system according to one embodiment of the present disclosure includes: a first acquisition unit that acquires a first dataset containing a plurality of data captured by a first shooting method and a second dataset containing a plurality of data captured by a second shooting method different from the first shooting method; a second acquisition unit that acquires an intermediate learning dataset from at least one of the first dataset and the second dataset based on a first feature included in a first feature group corresponding to the first dataset and a second feature included in a second feature group corresponding to the second dataset; and a learning unit that performs a first learning process on an inference model using the dataset included in the first dataset, performs a second learning process on a first trained model acquired by the first learning process using the intermediate learning dataset, and performs a third learning process on a second trained model acquired by the second learning process using the dataset included in the second dataset.

[0009] Further features of this disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings.

[0010] This shows the schematic configuration of the learning system for the inference model. This is a flowchart of the processing performed by the learning system according to the first embodiment. This shows an example of the configuration of the inference model according to the first embodiment. This shows an example of the distribution of the first and second datasets. This is a flowchart of the processing performed by the learning system according to the second embodiment. This is a flowchart of the processing performed by the learning system according to the third embodiment. This is a flowchart of the processing performed by the learning system according to the fourth embodiment. This shows the schematic configuration of the inference system according to the fifth embodiment. This is a flowchart of the processing performed by the inference device according to the fifth embodiment.

[0011] Embodiments of this disclosure will be described below with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed depending on the configuration of the device to which this disclosure applies or various conditions. This disclosure is not limited to the following embodiments and can be modified as appropriate without departing from its spirit. In addition, in the drawings described below, identical or functionally similar components are denoted by the same reference numerals and their descriptions may be omitted.

[0012] In the following, a machine learning model (inference model) refers to a learning model based on a machine learning algorithm. Specific machine learning algorithms include nearest neighbors, naive Bayes, decision trees, and support vector machines. Deep learning, which uses neural networks to generate features and joint weighting coefficients for learning, is also an example. Appropriate algorithms from the above can be used and applied to the following embodiments. Training data refers to learning data, which consists of pairs of input and output data. Ground truth data refers to the output data of the learning data (training data).

[0013] Furthermore, a pre-trained model refers to a machine learning model that follows any machine learning algorithm, such as deep learning, and has been trained (learned) in advance using appropriate training data. However, while a pre-trained model is obtained using appropriate training data in advance, it is not the case that it cannot be further trained; additional training can be performed. Additional training can be performed even after the device has been installed at the user's site. In addition, obtaining output data from input data using a pre-trained model is sometimes referred to as inference.

[0014] The learning system for the inference model according to the embodiment described below is a system that performs transfer learning using a first dataset for pre-training and a second dataset for additional training. In particular, it is characterized by obtaining an intermediate training dataset based on the similarity of features between the two datasets used for transfer learning and performing intermediate training. In the following, intermediate training refers to training performed between pre-training and additional training in order to reduce the differences in features between the datasets used for transfer learning.

[0015] <First Embodiment> The learning system and learning method for the inference model according to the first embodiment will be described below. The learning system for the inference model according to the first embodiment acquires data from a first dataset for pre-training that has features similar to those of a second dataset for additional training, and uses this data as a dataset for intermediate training.

[0016] The configuration and processing of the learning system for the inference model according to this embodiment will be described below with reference to Figures 1 to 4. Figure 1 is a block diagram showing the schematic configuration of the learning system for the inference model according to this embodiment. The learning system for the inference model 100 includes a storage unit 101, a first acquisition unit 102, a feature acquisition unit 103, a second acquisition unit 104, and a learning unit 105.

[0017] The memory unit 101 stores various datasets and inference models used for learning. The memory unit 101 can also store programs and other data that the learning system 100 uses to perform various processes.

[0018] The first acquisition unit 102 acquires the first dataset and the second dataset stored in the storage unit 101. The first acquisition unit 102 may also acquire the first dataset and the second dataset from an external device such as a server (not shown) connected to the learning system 100.

[0019] In this embodiment, we will describe a case where the first dataset and the second dataset are groups of images taken of the same body part using different imaging methods of the same type of modality. Specifically, the first dataset (group of mammography images) is acquired by the first imaging method (2D imaging) using a mammography device with a movable tube. The second dataset (group of tomosynthesis images) is acquired by the second imaging method (group of tomosynthesis images) using the same type of mammography device.

[0020] Conversely to this embodiment, a first dataset (tomosynthesis image group) may be acquired by a first imaging method (tomosynthesis imaging), and a second dataset (mammography image group) may be acquired by a second imaging method (2D imaging). Furthermore, the first and second datasets are not limited to the examples described above, and may be image groups acquired by different imaging methods, or image groups acquired by imaging methods targeting different body parts.

[0021] Furthermore, the first dataset may consist of images acquired with a first modality, and the second dataset may consist of images acquired with a second modality of a different type than the first modality. For example, the first modality may be an X-ray mammography system and the second modality may be a tomosynthesis system. Conversely, the first modality may be a tomosynthesis system and the second modality may be an X-ray mammography system. Moreover, the first and second modalities are not limited to an X-ray mammography system and a tomosynthesis system, but may also be an X-ray CT system, an MRI system, or a camera other than a medical imaging device, and do not necessarily have to be modalities that image the same body part.

[0022] The feature acquisition unit 103 acquires features from the first dataset and the second dataset. Details of the processing by the feature acquisition unit 103 will be explained in step S203.

[0023] The second acquisition unit 104 acquires an intermediate training dataset from the first dataset based on the similarity of the acquired features. Details of the processing by the second acquisition unit 104 will be explained in step S204.

[0024] The learning unit 105 performs a first learning process on the inference model using the first dataset and obtains a first trained model. Next, the learning unit 105 performs a second learning process on the first trained model using the intermediate learning dataset and obtains a second trained model. Then, the learning unit 105 performs a third learning process on the second trained model using the second dataset and obtains a third trained model. Details of the processing by the learning unit 105 will be explained in steps S205 to S207.

[0025] Here, the learning system 100 can be configured using a computer equipped with a processor and memory. The learning system 100 may be configured using a general-purpose computer, or a computer specifically designed for the learning system. Furthermore, the computer used may be, for example, a personal computer, a desktop PC, a notebook PC, or a tablet PC (portable information terminal). In addition, the learning system 100 may be configured using a cloud-type computer where some components are located on external devices.

[0026] Furthermore, each component of the learning system 100 other than the memory unit 101 may be composed of software modules executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit) or an FPGA (Field-Programmable Gate Array). In addition, each of these components may be composed of circuits that perform specific functions, such as an ASIC (Application Specific Integrated Circuit). The memory unit 101 may be composed of any storage medium, such as an optical disk such as a hard disk or memory.

[0027] Next, an example of the processing of the learning system 100 according to this embodiment will be explained using Figure 2. Figure 2 is a flowchart showing an example of the processing performed by the learning system 100.

[0028] (Step S201: Acquisition of the first dataset) In step S201, the first acquisition unit 102 acquires a group of mammography images, which is the first dataset, from the storage unit 101. The first acquisition unit 102 may also acquire the first dataset from an external device.

[0029] (Step S202: Acquisition of the second dataset) In step S202, the first acquisition unit 102 acquires the tomosynthesis image group, which is the second dataset, from the storage unit 101. The first acquisition unit 102 may also acquire the second dataset from an external device.

[0030] (Step S203: Acquisition of features) In step S203, the feature acquisition unit 103 acquires the feature quantities of each image included in the first dataset (mammography image group) and the second dataset (tomosynthesis image group). In this embodiment, the feature acquisition unit 103 acquires the contrast values ​​of the mammography images and tomosynthesis images as feature quantities.

[0031] However, the acquired features are not limited to these; for example, the maximum, minimum, mean, and median luminance values ​​that make up the image, SIFT features, HOG features, or intermediate features obtained from a dimensionality reduction algorithm may also be used. Furthermore, the acquired features do not have to be image features; they may also be values ​​associated with the image, such as age or breast density. Values ​​associated with the image may be acquired in conjunction with the image in steps S201 and S202. Moreover, the acquired features do not have to be a single value; they may be a combination of multiple features.

[0032] (Step S204: Obtaining an intermediate training dataset from the first dataset) In step S204, the second acquisition unit 104 obtains an intermediate training dataset from the first dataset based on the features (contrast values ​​in this embodiment) acquired in step S203.

[0033] First, the second acquisition unit 104 acquires the similarity between individual data (images) from the first dataset and the second dataset based on the acquired features. In this embodiment, the reciprocal of the distance between the features of the two datasets (e.g., the Euclidean distance) is defined as the similarity between the data. Note that the similarity may be defined not only as the reciprocal of the distance, but also as the distance minus a predetermined value (e.g., the maximum expected distance). Furthermore, the Mahalanobis distance or Manhattan distance may be used instead of the Euclidean distance.

[0034] When obtaining the similarity of a particular data point from the first dataset to the second dataset, the second acquisition unit 104 first calculates the sum of the distances between the feature quantities of the data point and the feature quantities of multiple data points that make up the second dataset. The second acquisition unit 104 defines the calculated sum as the distance of the data point to the second dataset. Note that the distance of the data point to the second dataset is not limited to the sum of the distances; statistical values ​​such as the mean or median of the distances may also be used. The second acquisition unit 104 then obtains the reciprocal of the distance of the data point to the second dataset as the similarity of the data point to the second dataset.

[0035] Furthermore, instead of calculating the distance on the features of the individual data that make up the second dataset, the distance may be calculated on statistical values ​​(measures of central tendency) such as the minimum, median, mean, or maximum of the features of multiple data that make up the second dataset.

[0036] Furthermore, any other method for obtaining the distance between the data of interest and the second dataset may be used. For example, a subspace of the second dataset may be obtained by reducing the dimensionality of the second dataset, and the reconstruction error when the data of interest is approximated by this subspace may be used as the distance. In this case, first, a network such as an autoencoder is prepared that is trained to take an image from the second dataset as input and reconstruct the input image from the second dataset. The second acquisition unit 104 can input the data of interest into this network and obtain the error (reconstruction error) between the image output from the network and the data of interest as the distance between the data of interest and the second dataset.

[0037] By performing the above process for each individual data point that makes up the first dataset, the second acquisition unit 104 can obtain the similarity of each individual data point that makes up the first dataset to the second dataset.

[0038] Next, the second acquisition unit 104 acquires intermediate training datasets from the first dataset based on the similarity to the second dataset. For example, the second acquisition unit 104 acquires a predetermined number of data as intermediate training datasets in descending order of similarity. Alternatively, instead of acquiring a predetermined number of data, the second acquisition unit 104 may acquire data with a similarity above a threshold as intermediate training datasets. Furthermore, the second acquisition unit 104 may combine these to acquire intermediate training datasets.

[0039] Furthermore, the second acquisition unit 104 may acquire intermediate training datasets from the first dataset based on distance (using proximity as a condition) without calculating similarity. For example, the second acquisition unit 104 may acquire a predetermined number of data as intermediate training datasets in order of proximity to the second dataset. Alternatively, the second acquisition unit 104 may acquire data with distances below a threshold as intermediate training datasets. Alternatively, the second acquisition unit 104 may combine these to acquire intermediate training datasets.

[0040] (Step S205: First Learning) In step S205, the learning unit 105 obtains an inference model from the storage unit 101, performs a first learning process using the first dataset, and obtains a first trained model. The learning unit 105 may also obtain the inference model from an external device such as a server (not shown) connected to the learning system 100.

[0041] In this embodiment, the inference model is a convolutional neural network (CNN). However, the inference model is not limited to this; other models such as DNN (Deep Neural Network), SVM (Support Vector Machine), and ViT (Vision Transformer) may also be used.

[0042] Here, with reference to Figure 3, an example of the CNN configuration related to the inference model according to this embodiment will be described. However, the CNN configuration is not limited to that described below and may be arbitrarily changed according to the desired configuration. Figure 3 shows an example of the configuration of the inference model. The configuration shown in Figure 3 consists of a group of layers that are responsible for processing the input value group and outputting the result. The types of layers included in this configuration are, as shown in Figure 3, a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.

[0043] The convolutional layer is a layer that performs a convolution process on a group of input values according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. Note that the dimensionality of the filter kernel size may be changed according to the dimensionality of the input image.

[0044] The downsampling layer is a layer that performs a process of reducing the number of output values to be less than the number of input values by thinning out or synthesizing the group of input values. Specifically, as such a process, for example, there is Max Pooling processing.

[0045] The upsampling layer is a layer that performs a process of increasing the number of output values to be more than the number of input values by duplicating the group of input values or adding interpolated values from the group of input values. Specifically, as such a process, for example, there is linear interpolation processing.

[0046] The synthesis layer is a layer that inputs groups of values such as the group of output values of a certain layer or the group of pixel values constituting an image from a plurality of sources, and performs a process of synthesizing them by concatenating or adding them.

[0047] In such a configuration, the group of pixel values constituting the input image Im301 and the group of values output after passing through the convolution processing block are synthesized in the synthesis layer. Thereafter, the synthesized group of pixel values is shaped into the inference result 302 in the last convolutional layer.

[0048] Note that when the parameter settings for the layer group and node group constituting the neural network are different, the degree of reproducibility of the tendency trained from the learning data may be different during inference, so care is required. That is, in many cases, appropriate parameters are different according to the form implemented, so they can be changed to preferable values as necessary.

[0049] In addition to changing the parameters as described above, CNNs can sometimes achieve better characteristics by changing their configuration. These better characteristics include, for example, producing more accurate inference results, shorter processing times, and shorter training times for machine learning models.

[0050] The CNN configuration is a U-net type machine learning model that has the functionality of an encoder consisting of multiple layers including multiple downsampling layers, and the functionality of a decoder consisting of multiple layers including multiple upsampling layers. In other words, the CNN configuration includes a U-shaped structure having encoder and decoder functions. In the U-net type machine learning model, the positional information (spatial information) that has been obscured in the multiple layers configured as the encoder is configured to be usable in the multiple layers configured as the decoder in layers of the same dimension (layers corresponding to each other) (for example, by using skip connections).

[0051] Although not shown in the diagram, examples of modifications to the CNN configuration include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolutional layer. Alternatively, a fully connected layer or an output layer may be incorporated after the convolutional layer.

[0052] Here, the GPU can perform calculations efficiently by processing more data in parallel. Therefore, when performing learning multiple times using machine learning algorithms such as deep learning, it is effective to perform the processing on the GPU. Accordingly, in this embodiment, the learning unit 105 uses a GPU in addition to the CPU for processing. Specifically, when executing a learning program that includes a learning model, learning is performed by the CPU and GPU working together to perform calculations. Note that the processing in the learning unit 105 may be performed by the CPU or the GPU alone.

[0053] The learning unit 105 may also include an error detection unit and an update unit (not shown). The error detection unit obtains the error between the output data output from the output layer of the neural network and the correct data, according to the input data input to the input layer. The error detection unit may calculate the error between the output data from the neural network and the correct data using a loss function. The update unit updates the connection weight coefficients between nodes of the neural network, etc., based on the error obtained by the error detection unit, so as to reduce the error. This update unit updates the connection weight coefficients, etc., for example, using backpropagation. Backpropagation is a method of adjusting the connection weight coefficients, etc., between nodes of each neural network so as to reduce the above error.

[0054] (Step S206: Second Learning) In step S206, the learning unit 105 performs a second learning process on the first trained model, which underwent the first learning process in step S205, using the intermediate learning dataset obtained in step S204, and obtains a second trained model.

[0055] (Step S207: Third Learning) In step S207, the learning unit 105 performs a third learning process on the second trained model, which underwent the second learning process in step S206, using the second dataset, and obtains a third trained model.

[0056] As described above, the learning system 100 according to this embodiment comprises a first acquisition unit 102, a second acquisition unit 104, and a learning unit 105. The first acquisition unit 102 functions as an example of a first acquisition unit that acquires a first dataset containing a plurality of data captured by a first shooting method, and a second dataset containing a plurality of data captured by a second shooting method different from the first shooting method. The second acquisition unit 104 functions as an example of a second acquisition unit that acquires an intermediate learning dataset from the first dataset based on a first feature quantity included in a first feature quantity group corresponding to the first dataset and a second feature quantity included in a second feature quantity group corresponding to the second dataset. The learning unit 105 functions as an example of a learning unit that performs first to third learning processing. The learning unit 105 performs a first learning processing on the inference model using the first dataset. The learning unit 105 also performs a second learning processing on the first trained model acquired by the first learning processing using the intermediate learning dataset. Furthermore, the learning unit 105 performs a third learning process on the second trained model obtained by the second learning process, using the second dataset.

[0057] With this configuration, the learning system 100 according to this embodiment can obtain a dataset suitable for intermediate learning from the first dataset without obtaining an intermediate learning dataset from an external source. This makes it possible to efficiently perform transfer learning even when there is a large difference in the features between the first dataset and the second dataset.

[0058] The first imaging method is a method for capturing medical images, and the second imaging method may be a method for capturing medical images that is different from the first imaging method, but targets the same body part as the first imaging method. For example, the first imaging method may be either X-ray mammography or tomosynthesis, and the second imaging method may be the other of X-ray mammography or tomosynthesis.

[0059] Furthermore, the feature quantities used by the second acquisition unit 104 can be feature quantities obtained from the image. These feature quantities may be, for example, the contrast value of the image, the maximum, minimum, average, and median values ​​of the brightness values ​​that constitute the image, SIFT features, HOG features, or intermediate features obtained from a dimensionality reduction algorithm.

[0060] The second acquisition unit 104 can acquire an intermediate training dataset based on the similarity or distance between the first feature and the second feature. More specifically, the second acquisition unit 104 can acquire an intermediate training dataset by a first process that acquires data from the first dataset that has first features similar to the second set of features.

[0061] For example, the second acquisition unit 104 can acquire a predetermined number of data from the first dataset as an intermediate training dataset, sorted by similarity or distance. Alternatively, the second acquisition unit 104 may acquire data from the first dataset as an intermediate training dataset if the similarity is above a threshold or the distance is below a threshold.

[0062] The second dataset may be acquired using a different modality than the one used to acquire the first dataset. Conversely, the first and second datasets may be acquired using the same modality.

[0063] In step S203, the feature acquisition unit 103 may set an ROI (Region of Interest) for each image included in the first dataset and the second dataset, and acquire features within the ROI.

[0064] <Modification 1 of the First Embodiment> In the first embodiment, the second acquisition unit 104 acquired the intermediate learning dataset from the first dataset, but the method of acquiring the intermediate learning dataset is not limited to this. The second acquisition unit 104 may acquire the intermediate learning dataset from the second dataset. For example, if the feature distribution of the first dataset is narrow and the feature distribution of the second dataset is wide, the second acquisition unit 104 may acquire the intermediate learning dataset from the second dataset instead of the first dataset based on the similarity of the features. In this case, the processing for the first dataset in step S204 may be replaced with processing for the second dataset, and the processing for the second dataset may be replaced with processing for the first dataset.

[0065] The second acquisition unit 104 may also acquire an intermediate training dataset from the second dataset based on the distance between the features.

[0066] As described above, the second acquisition unit 104 according to the modified example 1 of the first embodiment can function as an example of a second acquisition unit that acquires an intermediate learning dataset from a second dataset based on a first feature group included in a first feature group corresponding to a first dataset and a second feature group included in a second feature group corresponding to a second dataset.

[0067] According to this modified example, a dataset suitable for intermediate learning can be obtained from the second dataset without having to acquire an intermediate learning dataset from an external source. This allows for efficient transfer learning even when there is a large difference in the features between the first and second datasets.

[0068] Furthermore, the second acquisition unit 104 in this modified example can acquire an intermediate learning dataset by a second process that acquires data having a second feature similar to the first feature group from the second dataset. For example, the second acquisition unit 104 can acquire a predetermined number of data from the second dataset as an intermediate learning dataset in order of high similarity or close distance. Alternatively, the second acquisition unit 104 may acquire data from the second dataset as an intermediate learning dataset if the similarity is above a threshold or the distance is below a threshold.

[0069] Furthermore, the decision of whether to obtain the intermediate training dataset from the first dataset or the second dataset may be made based on the breadth of the feature distribution. For example, if the second dataset has a wider feature distribution, the intermediate training dataset may be obtained from the second dataset. Alternatively, instead of the breadth of the feature distribution, the dataset from which to obtain the intermediate training dataset may be determined based on the amount of data in the first and second datasets. For example, if the second dataset has more data, the intermediate training dataset may be obtained from the second dataset. Therefore, the second acquisition unit 104 can select a dataset from which to obtain the intermediate training dataset based on the breadth of the feature distribution or the amount of data in the first and second datasets.

[0070] <Modification 2 of the First Embodiment> In the first embodiment, the second acquisition unit 104 acquired the intermediate training dataset from the first dataset, but the method of acquiring the intermediate training dataset is not limited to this. The second acquisition unit 104 may acquire the intermediate training dataset from both the first dataset and the second dataset. In this case, the second acquisition unit 104 can acquire the intermediate training dataset from the first dataset and the second dataset based on the similarity of the features.

[0071] Figure 4 is a schematic diagram plotting the first dataset, the second dataset, and the intermediate learning dataset 401 on two feature axes. In the modified example 2 of the first embodiment, the second acquisition unit 104 can acquire data from the first dataset and the second dataset that have similar features to each other as the intermediate learning dataset 401. In this case, the processing for the first dataset in step S204 should be replaced with processing for both the first dataset and the second dataset.

[0072] The second acquisition unit 104 may acquire a predetermined number of data from the first dataset and the second dataset in descending order of similarity as intermediate training datasets. Alternatively, the second acquisition unit 104 may acquire a predetermined number of data from the first dataset and the second dataset, respectively, in descending order of similarity as intermediate training datasets. In this case, the number of data acquired from the first dataset and the second dataset may be the same or different.

[0073] Furthermore, the second acquisition unit 104 may acquire an intermediate training dataset from the first dataset and the second dataset based on the distance between the features.

[0074] As described above, the second acquisition unit 104 according to this modified example can function as an example of a second acquisition unit that acquires an intermediate learning dataset from the first dataset and the second dataset based on the first feature included in the first feature group corresponding to the first dataset and the second feature included in the second feature group corresponding to the second dataset.

[0075] According to this modified example, when there is a large difference between the feature distribution of the first dataset and the feature distribution of the second dataset, the diversity of data used for intermediate training can be ensured by obtaining intermediate training data from both the first and second datasets.

[0076] Furthermore, the second acquisition unit 104 in this modified example can acquire an intermediate training dataset based on the similarity or distance between the first feature and the second feature group, and the similarity or distance between the second feature and the first feature group. Specifically, the second acquisition unit 104 can acquire an intermediate training dataset by performing a first process of acquiring data from the first dataset that has first features similar to the second feature group, and a second process of acquiring data from the second dataset that has second features similar to the first feature group.

[0077] For example, the second acquisition unit 104 can acquire a predetermined number of data from the first and second datasets as intermediate learning datasets, sorted by similarity or distance. Alternatively, the second acquisition unit 104 may acquire data from the first and second datasets as intermediate learning datasets if the similarity is above a threshold or the distance is below a threshold.

[0078] <Second Embodiment> In the first embodiment, an intermediate learning dataset was obtained from the first dataset based on the similarity of features. In contrast, in the second embodiment, an intermediate learning dataset is obtained from the first dataset based on the proportion of datasets having predetermined features. In the following description, the same reference numerals are used for the same configurations and processes as in the first embodiment, and their explanations are omitted. The schematic configuration of the learning system for the inference model according to this embodiment is the same as that shown in Figure 1, and the flow of the processes to be executed is shown in Figure 5. The second acquisition unit 104, step S203, and step S501, which differ in processing content from the first embodiment, will be described below.

[0079] The second acquisition unit 104 acquires an intermediate training dataset from the first dataset using the proportion of first features constituting the first feature group corresponding to the first dataset and the proportion of second features constituting the second feature group corresponding to the second dataset. Details will be explained in steps S203 and S501.

[0080] Next, an example of the processing of the learning system 100 according to this embodiment will be explained using the flowchart shown in Figure 5. Figure 5 is a flowchart showing an example of the processing performed by the learning system 100 according to this embodiment. The processing in steps S201 and S202 is the same as the processing in steps S201 and S202 according to the first embodiment, so the explanation will be omitted. When the second dataset is acquired in step S202, the processing moves to step S203.

[0081] (Step S203: Feature acquisition) In step S203, the feature acquisition unit 103 acquires a first feature from the first dataset and a second feature from the second dataset. In this embodiment, each feature is a categorical value indicating whether the shape of the lesion in the image is circular or non-circular. However, the features are not limited to these, and may also be the outline of the lesion in the image or whether the lesion is benign or malignant. Furthermore, the acquired features are not limited to features related to lesions, but may also be the manufacturer of the modality used for imaging, the medical institution that took the image, or the race of the patient. Alternatively, the same features as in the first embodiment may be acquired. Moreover, multiple features may be combined. Once the features are acquired in step S203, the process moves to step S501.

[0082] (Step S501: Obtaining an intermediate training dataset from the proportion of features constituting the dataset) In step S501, the second acquisition unit 104 obtains an intermediate training dataset from the first dataset based on the proportion of the first features and the proportion of the second features obtained in step S203. Specifically, the second acquisition unit 104 obtains data from the first dataset such that the proportion of data where the lesion shape is circular (the proportion of the third feature) constituting the intermediate training dataset is an intermediate value (for example, the mean) between the proportion of data where the lesion shape is circular constituting the first dataset (the proportion of the first feature) and the proportion of data where the lesion shape is circular constituting the second dataset (the proportion of the second feature).

[0083] For example, if 30% of the first dataset consists of circular lesions and 50% of the second dataset consists of circular lesions, the second acquisition unit 104 randomly acquires data from the first dataset, including both circular and non-circular lesions, so that the proportion of circular lesions in the intermediate training dataset becomes 40%. However, the second acquisition unit 104 may also acquire data from the first dataset so that the proportion of circular lesions is approximately the same as the proportion of circular lesions in the second dataset, thereby acquiring the intermediate training dataset.

[0084] As described above, the second acquisition unit 104 in this embodiment can acquire an intermediate training dataset based on the proportion of first features included in the first feature group and the proportion of second features included in the second feature group. More specifically, the second acquisition unit 104 can acquire an intermediate training dataset from the first dataset such that the proportion of third features included in the third feature group corresponding to the intermediate training dataset is between the proportion of first features included in the first feature group and the proportion of second features included in the second feature group.

[0085] According to this embodiment, without obtaining an intermediate learning dataset from an external source, a dataset suitable for intermediate learning can be obtained from the first dataset and the second dataset, selected based on the proportion of features constituting the datasets. This makes it possible to efficiently perform transfer learning even when there is a large difference in the proportion of features constituting the first dataset and the second dataset.

[0086] <Modification 1 of the Second Embodiment> In the second embodiment, the second acquisition unit 104 acquired the intermediate training dataset from the first dataset, but the method of acquiring the intermediate training dataset is not limited to this. The second acquisition unit 104 may acquire the intermediate training dataset from the second dataset, or from the first dataset and the second dataset.

[0087] When the number of data points containing a specific feature that make up the first dataset is small, it is not possible to obtain a sufficient number of data points containing that feature as intermediate training data. In contrast, according to this modified example, an intermediate training dataset is obtained from the second dataset, or from both the first and second datasets. This ensures that even if there are few data points containing a specific feature in the first dataset, a sufficient amount of data for intermediate training can be secured.

[0088] The process for obtaining the intermediate training dataset may be the same as the process in step S501, except that the intermediate training dataset is obtained from the second dataset, or from the first dataset and the second dataset. For example, the second acquisition unit 104 can obtain the intermediate training dataset from the second dataset such that the proportion of the third feature included in the third feature group corresponding to the intermediate training dataset is between the proportion of the first feature included in the first feature group and the proportion of the second feature included in the second feature group. Alternatively, the second acquisition unit 104 may obtain the intermediate training dataset from the first dataset and the second dataset such that the proportion of the third feature included in the third feature group corresponding to the intermediate training dataset is between the proportion of the first feature included in the first feature group and the proportion of the second feature included in the second feature group.

[0089] <Third Embodiment> In the first embodiment, one dataset for intermediate learning was acquired. In contrast, in this embodiment, multiple datasets for intermediate learning are acquired. In the following description, the same reference numerals are used for components and processes as in the first embodiment, and their descriptions are omitted. The schematic configuration of the learning system for the inference model according to this embodiment is the same as that shown in Figure 1, and the execution flow is shown in Figure 6. The second acquisition unit 104, learning unit 105, step S601, and step S602, which differ in processing content from the first embodiment, will be described below.

[0090] The second acquisition unit 104 acquires multiple intermediate training datasets from the first dataset based on the similarity of the features. Details will be explained in step S601.

[0091] The learning unit 105 performs a first learning process on the inference model using the first dataset and obtains a first trained model. Next, the learning unit 105 repeatedly performs a second learning process on the first trained model using multiple intermediate learning datasets and obtains a second trained model. Then, the learning unit 105 performs a third learning process on the second trained model using the second dataset and obtains a third trained model. Details will be explained in steps S206 and S602.

[0092] Next, an example of the processing of the learning system 100 according to this embodiment will be explained using the flowchart shown in Figure 6. Figure 6 is a flowchart of an example of the processing performed by the learning system 100 according to this embodiment. The processing in steps S201 to S203 is the same as the processing in steps S201 to S203 according to the first embodiment, so the explanation will be omitted. When the feature quantities are acquired in step S203, the processing moves on to step S601.

[0093] (Step S601: Obtaining multiple intermediate training datasets from the first dataset) In step S601, the second acquisition unit 104 obtains multiple intermediate training datasets from the first dataset based on the similarity of the contrast values ​​obtained in step S203. For example, the second acquisition unit 104 can obtain multiple intermediate training datasets by repeatedly acquiring a predetermined number of data from the first dataset in descending order of similarity.

[0094] Alternatively, multiple thresholds may be set for similarity, and the second acquisition unit 104 may acquire data with similarity equal to or greater than each threshold as a single intermediate learning dataset, thereby acquiring multiple intermediate learning datasets.

[0095] Furthermore, a method of acquiring a predetermined number of data as a single intermediate learning dataset may be combined with a method of acquiring data with a similarity above a threshold as a single intermediate learning dataset. For example, the second acquisition unit 104 may repeatedly acquire a predetermined number of data from among multiple data with a similarity above a threshold, in descending order of similarity, to acquire multiple intermediate learning datasets. Note that the data constituting an intermediate learning dataset may overlap with the data constituting other intermediate learning datasets.

[0096] Furthermore, the second acquisition unit 104 may acquire multiple intermediate training datasets from the first dataset based on the distance between the features.

[0097] Once multiple intermediate training datasets are obtained in step S601, the process moves to step S205. In step S205, the first training is performed as in the first embodiment, and then the process moves to step S206.

[0098] (Step S206: Second Learning) In step S206, the learning unit 105 performs a second learning operation on the first trained model, which underwent the learning process in step S205, using one of the multiple intermediate learning datasets obtained in step S601, and obtains a second trained model. Once the second learning operation is complete, the process proceeds to step S602.

[0099] (Step S602: Determining whether there are unused intermediate training datasets) In step S602, the learning unit 105 determines whether there are unused intermediate training datasets. If it is determined in step S602 that there are unused intermediate training datasets, the process moves to step S206, and the learning unit 105 performs a second training on the already acquired second trained model using the unused intermediate training datasets. On the other hand, if it is determined in step S602 that there are no unused intermediate training datasets, the process moves to step S207, and a third training is performed in the same manner as in the first embodiment.

[0100] The order in which the intermediate training datasets are used for the second training can be arbitrary. For example, the intermediate training datasets may be used in order of decreasing similarity, or in order of decreasing similarity. Alternatively, the intermediate training datasets may be used in order of decreasing similarity.

[0101] As described above, the second acquisition unit 104 in this embodiment can acquire a plurality of intermediate learning datasets. The second acquisition unit 104 can acquire intermediate learning datasets based, for example, on the similarity or distance between the first feature and the second feature. Furthermore, the learning unit 105 can perform the second learning process step by step using the plurality of intermediate learning datasets.

[0102] As described above, according to this embodiment, stepwise intermediate learning is performed using different intermediate learning datasets. This allows for efficient transfer learning even when there is a large difference in features between a single intermediate learning dataset and the first and second datasets, in addition to the effects of the first embodiment.

[0103] <Modification 1 of the Third Embodiment> In the third embodiment, the second acquisition unit 104 acquired the intermediate learning dataset from the first dataset, but the method of acquiring the intermediate learning dataset is not limited to this. The second acquisition unit 104 may acquire the intermediate learning dataset from the second dataset, or from the first dataset and the second dataset. In this case, the process for acquiring the intermediate learning dataset may be the same as the process described in Modifications 1 and 2 of the first embodiment. In this modified configuration as well, stepwise intermediate learning is performed using different intermediate learning datasets. As a result, in addition to the effects of the modified configuration of the first embodiment, transfer learning can be efficiently performed even when there is a large difference in the features of one intermediate learning dataset compared to the first and second datasets.

[0104] <Fourth Embodiment> In the second embodiment, one dataset for intermediate learning was acquired. In contrast, in the fourth embodiment, multiple datasets for intermediate learning are acquired. In the following description, the same reference numerals are used for components and processes as in the second embodiment, and their explanations are omitted. The schematic configuration of the learning system for the inference model according to this embodiment is the same as that shown in Figure 1, and the execution flow is shown in Figure 7. The second acquisition unit 104, the learning unit 105, step S701, and step S702, which differ in processing content from the second embodiment, will be described below.

[0105] The second acquisition unit 104 acquires multiple intermediate training datasets from the first dataset using the proportion of first features constituting the first feature group corresponding to the first dataset and the proportion of second features constituting the second feature group corresponding to the second dataset. Details will be explained in step S701.

[0106] The learning unit 105 performs a first learning process on the inference model using the first dataset and obtains a first trained model. Next, the learning unit 105 repeatedly performs a second learning process on the first trained model using multiple intermediate learning datasets and obtains a second trained model. Then, the learning unit 105 performs a third learning process on the second trained model using the second dataset and obtains a third trained model. Details will be explained in steps S206 and S702.

[0107] Next, an example of the processing of the learning system 100 according to this embodiment will be explained using the flowchart shown in Figure 7. Figure 7 is a flowchart of an example of the processing performed by the learning system 100 according to this embodiment. The processing in steps S201 to S203 is the same as the processing in steps S201 to S203 according to the second embodiment, so the explanation will be omitted. When the first and second feature quantities are acquired in step S203, the processing moves to step S701.

[0108] (Step S701: Obtaining intermediate training datasets from the proportion of features constituting the dataset) In step S701, the second acquisition unit 104 obtains multiple intermediate training datasets from the first dataset based on the proportion of the first feature and the proportion of the second feature obtained in step S203. Specifically, the second acquisition unit 104 obtains data from the first dataset such that the proportion of data where the lesion shape is circular (the proportion of the third feature) constituting the intermediate training dataset is between the proportion of data where the lesion shape is circular constituting the first dataset (the proportion of the first feature) and the proportion of data where the lesion shape is circular constituting the second dataset (the proportion of the second feature).

[0109] For example, suppose 30% of the first dataset consists of circular lesions and 60% of the second dataset consists of circular lesions. In this case, the second acquisition unit 104 randomly acquires data of circular and non-circular lesions from the first dataset so that the proportion of circular lesions in the first intermediate training dataset becomes 40%. The second acquisition unit 104 also randomly acquires data of circular and non-circular lesions from the first dataset so that the proportion of circular lesions in the second intermediate training dataset becomes 50%. Note that more than three intermediate training datasets may be acquired. Also, the data constituting an intermediate training dataset may overlap with the data constituting other intermediate training datasets.

[0110] Once multiple intermediate training datasets are obtained in step S701, the process moves to step S205. In step S205, the first training is performed as in the second embodiment, and then the process moves to step S206.

[0111] (Step S206: Second Learning) In step S206, the learning unit 105 performs a second learning operation on the first trained model, which underwent training in step S205, using one of the multiple intermediate learning datasets obtained in step S701, and obtains a second trained model. Once the second learning operation is complete, the process moves on to step S702.

[0112] (Step S702: Determining whether there are unused intermediate training datasets) In step S702, the learning unit 105 determines whether there are unused intermediate training datasets. If it is determined in step S702 that there are unused intermediate training datasets, the process moves to step S206, and the learning unit 105 performs a second training on the already acquired second trained model using the unused intermediate training datasets. On the other hand, if it is determined in step S702 that there are no unused intermediate training datasets, the process moves to step S207, and a third training is performed in the same manner as in the second embodiment.

[0113] The order in which the intermediate training datasets are used for the second training can be arbitrary. For example, the intermediate training datasets may be used in the second training in order of the proportion of features being closest to the proportion of features in the second training dataset, or they may be used in order of the proportion of features being furthest from the proportion of features in the second training dataset. Alternatively, the intermediate training datasets may be used in the second training in an order unrelated to the proximity of the proportion of features.

[0114] As described above, the second acquisition unit 104 in this embodiment can acquire a plurality of intermediate learning datasets. The second acquisition unit 104 can acquire intermediate learning datasets based, for example, on the proportion of a first feature included in the first feature group and the proportion of a second feature included in the second feature group. Furthermore, the learning unit 105 can perform a second learning process step by step using the plurality of intermediate learning datasets.

[0115] As described above, according to this embodiment, stepwise intermediate learning is performed using different intermediate learning datasets. This allows for efficient transfer learning even when there is a large difference in features between a single intermediate learning dataset and the first and second datasets, in addition to the effects of the second embodiment.

[0116] <Modification 1 of the Fourth Embodiment> In the fourth embodiment, the second acquisition unit 104 acquired the intermediate learning dataset from the first dataset, but the method of acquiring the intermediate learning dataset is not limited to this. The second acquisition unit 104 may acquire the intermediate learning dataset from the second dataset, or from the first dataset and the second dataset. In this case, the process for acquiring the intermediate learning dataset may be the same as the process described in Modification 1 of the second embodiment. In this modified configuration as well, stepwise intermediate learning is performed using different intermediate learning datasets. As a result, in addition to the effects of the second embodiment, transfer learning can be efficiently performed even when there is a large difference in the features of one intermediate learning dataset compared to the first and second datasets.

[0117] <Fifth Embodiment> In the fifth embodiment of this disclosure, an inference system, inference device, and inference method that acquire inference results based on an input image using a trained model trained by a learning system according to any of the embodiments and modifications described above will be described. The inference system according to this embodiment will be described below with reference to Figures 8 and 9. Figure 8 is a block diagram showing the schematic configuration of the inference system 800 according to this embodiment.

[0118] The inference system 800 includes a shooting device 810, an inference device 820, a display unit 830, and an input unit 840. The shooting device 810 takes a photograph of the subject and acquires an input image.

[0119] The inference device 820 is communicatively connected to the imaging device 810, the display unit 830, and the input unit 840. The inference device 820 uses a trained model to infer the presence or absence of lesions based on the input image acquired by the imaging device 810. The inference device 820 is equipped with a storage unit 821, an acquisition unit 822, an inference unit 823, and a display control unit 824.

[0120] The memory unit 821 stores the trained model trained by the learning system described above, input images acquired from the imaging device 810, etc., and inference results inferred by the inference unit 823. The memory unit 821 may also include programs for the inference device 820 to perform various processes.

[0121] The acquisition unit 822 can acquire input images captured by the imaging device 810. The acquisition unit 822 can also acquire input images from an external device (not shown) connected to the inference device 820 via a network such as the Internet.

[0122] The inference unit 823 performs inference processing using the trained model and infers the presence or absence of a lesion (tumor) based on the input image acquired by the acquisition unit 822. The inference unit 823 can also perform analysis processing on the inference result. The input image should correspond to the data that makes up the second dataset used to train the trained model. Hereafter, the input image will be described as a tomosynthesis image.

[0123] The display control unit 824 controls the display of the display unit 830. The display control unit 824 can display, for example, information about the subject, information about the images taken, and various acquired images on the display unit 830.

[0124] Here, the inference device 820 can be configured using a computer equipped with a processor and memory. The inference device 820 may be configured using a general-purpose computer, or a computer dedicated to the inference system. Furthermore, the inference device 820 may be configured using, for example, a personal computer, such as a desktop PC, notebook PC, or tablet PC (portable information terminal). In addition, the inference device 820 may be configured using a cloud-type computer where some components are located on external devices.

[0125] Furthermore, each component of the inference device 820 other than the storage unit 821 may be composed of software modules executed by a processor such as a CPU or MPU. The processor may be, for example, a GPU or FPGA. In addition, each of these components may be composed of circuits that perform specific functions, such as ASICs. The storage unit 821 may be composed of any storage medium, such as an optical disk such as a hard disk or memory.

[0126] The display unit 830 is configured with any monitor and displays various information such as subject information, various images, and a mouse cursor according to the operation of the input unit 840, in accordance with the control of the display control unit 824. The input unit 840 is an input device that gives instructions to the inference device 820, and specifically includes a keyboard or mouse. In this embodiment, the inference device 820, the display unit 830, and the input unit 840 are configured separately, but they or parts thereof may be configured as an integral unit. For example, the display unit 830 may be configured with a touch panel display, in which case the display unit 830 can also be used as the input unit 840.

[0127] The processes executed by the inference device 820 according to this embodiment will now be described with reference to Figure 9. Figure 9 is a flowchart of the processes executed by the inference device 820.

[0128] In step S901, the acquisition unit 822 acquires an input image from the imaging device 810. The acquisition unit 822 may also acquire the input image from an external device such as a server (not shown) connected to the inference device 820.

[0129] In step S902, the inference unit 823 inputs the input image to the trained model and obtains the inference result output from the trained model. In this embodiment, the inference unit 823 obtains the inference result from the trained model regarding the presence or absence of lesions in the input image.

[0130] Here, the trained model can be a trained model that has been trained using training data in which images for which the presence or absence of lesions should be inferred are used as input data, and information indicating the presence or absence of lesions in said images is used as output data. For this reason, in the training process according to the above-described embodiment and modified example, transfer learning can be performed using the first dataset, the intermediate training dataset, and the second dataset as input data for the training data.

[0131] More specifically, the first training may be performed using training data consisting of images constituting the first dataset and information indicating the presence or absence of lesions in those images as output data. Furthermore, the second training may be performed using images constituting an intermediate training dataset and training data consisting of information indicating the presence or absence of lesions in those images as output data. Similarly, the third training may be performed using images constituting the second dataset and training data consisting of information indicating the presence or absence of lesions in those images as output data.

[0132] Furthermore, the information used as output data for training data, which indicates the presence or absence of lesions in the input images, may be information determined by a physician or other medical professional, or it may be information obtained through arbitrary rule-based processing.

[0133] In step S903, the display control unit 824 displays the inference results obtained in step S902 on the display unit 830. Alternatively, the display control unit 824 may display the analysis results obtained by the inference unit 823, which performs arbitrary analysis processing on the inference results obtained in step S902, on the display unit 830. Furthermore, the inference device 820 may transmit the inference results obtained in step S902 and the analysis results of those inference results to an external device.

[0134] As described above, the inference system 800 according to this embodiment includes an imaging device 810 for photographing a subject and an inference device 820 that is communicatively connected to the imaging device 810. The inference device 820 includes an acquisition unit 822 that functions as an example of an acquisition unit for acquiring images, and an inference unit 823 that functions as an example of an inference unit for acquiring inference results by inputting images into a trained model. Here, the trained model is a trained model that has undergone first to third training processes. The first training process is performed on the inference model using a first dataset containing a plurality of data captured using a first imaging method. The second training process is performed on the first trained model acquired by the first training process using an intermediate training dataset. The third training process is performed on the second trained model acquired by the second training process using a second dataset containing a plurality of data captured using a second imaging method different from the first imaging method. Furthermore, the intermediate training dataset is obtained from at least one of the first and second datasets, based on the first features included in the first feature group corresponding to the first dataset and the second features included in the second feature group corresponding to the second dataset.

[0135] With this configuration, the inference system 800 according to this embodiment can perform inference processing using a trained model that has undergone efficient transfer learning, even when there is a large difference in the features between the first dataset and the second dataset. Therefore, even when it is difficult to prepare the dataset targeted by the inference model, transfer learning can be efficiently performed using another dataset with a large difference in features, and highly accurate inference processing can be performed using the trained model.

[0136] In this embodiment, the inference unit 823 is configured to use a trained model stored in the memory unit 821. However, the trained model may be provided on an external device such as a server connected to the inference device 820. In this case, the inference unit 823 can access the external device and use the trained model.

[0137] Furthermore, the inference device 820 may further include the configuration of the learning system 100. In this case, the inference device 820 can perform learning according to any of the embodiments and modifications described above and obtain a trained model.

[0138] In this embodiment, the inference process is described as inferring the presence or absence of a lesion, but the inference process by the inference unit 823 is not limited to this. The inference process by the inference unit 823 may include, for example, a process for detecting abnormal areas in the input image, a process for classifying abnormal areas in the input image, a process for analyzing the input image, a process for generating another type of image from the input image, a process for improving the image quality of the input image, and so on.

[0139] The training of a pre-trained model should be performed for the processes corresponding to the inference process. For example, if the inference process is to detect abnormal areas in an input image, training should be performed using training data in which the image to be abnormalized is used as input data and information indicating the abnormal areas in that image is used as output data. As information indicating the abnormal areas, for example, a labeled image with labels indicating each region in the image may be used.

[0140] Furthermore, for example, if the inference process is a process of classifying abnormal areas in an input image, training can be performed using training data in which the image to be abnormalized is used as input data and the information obtained by classifying the abnormal areas in that image is used as output data. The information used to classify the abnormal areas may include, for example, information indicating whether the lesion is benign or malignant.

[0141] Similarly, if the inference process is a process that analyzes an input image, training can be performed using training data in which the image to be analyzed is used as input data and the analysis result of that image using any method is used as output data. Furthermore, if the inference process is a process that generates another type of image from an input image, training can be performed using training data in which the original image is used as input data and the other type of image to be generated is used as output data.In addition, if the inference process is a process that enhances the image quality of an input image, training can be performed using training data in which the image to be enhanced is used as input data and the image after any enhancement process has been applied to that image is used as output data.

[0142] Regardless of which inference process is performed, transfer learning is carried out using the first dataset, the intermediate learning dataset, and the second dataset as described in the embodiments and modifications described above. This allows for efficient transfer learning even when there is a large difference in the features between the first dataset and the second dataset.

[0143] In addition, in the first and third learning processes described in the above embodiments and modifications, the learning process may be performed while omitting data included in the intermediate learning dataset, for example, in order to avoid duplicate learning of data included in the intermediate learning dataset. For this reason, the learning unit 105 may perform the first learning process using a dataset included in the first dataset other than the data included in the intermediate learning dataset. Furthermore, the learning unit 105 may perform the second learning process using a dataset included in the second dataset other than the data included in the intermediate learning dataset.

[0144] Furthermore, in the trained models described in the above embodiments and modifications, it is thought that the magnitude of the brightness values ​​of the input image, the order and slope of the bright and dark areas, their positions, distributions, and continuity are extracted as part of the features and used in the inference process.

[0145] (Other Embodiments) The present disclosure can also be realized by supplying a program that implements one or more functions of the embodiments and modifications described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. The computer may have one or more processors or circuits and may include a plurality of separate computers or a network of a plurality of separate processors or circuits for reading and executing computer executable instructions.

[0146] A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). Alternatively, a processor or circuit may include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0147] According to the above embodiment and its modifications, when performing transfer learning using a pre-training dataset and an additional training dataset, transfer learning can be performed efficiently even when there is a large difference in the features depicted in the two datasets.

[0148] This disclosure is not limited to the embodiments described above, and various modifications and alterations are possible without departing from the spirit and scope of this disclosure. Accordingly, the following claims are attached to make the scope of this disclosure public.

[0149] This application claims priority based on Japanese Patent Application No. 2024-156594, filed on September 10, 2024, and all of its contents are incorporated herein by reference.

Claims

1. A learning system for an inference model comprising: a first acquisition unit that acquires a first dataset containing a plurality of data captured by a first shooting method and a second dataset containing a plurality of data captured by a second shooting method different from the first shooting method; a second acquisition unit that acquires an intermediate learning dataset from at least one of the first dataset and the second dataset based on a first feature included in a first feature group corresponding to the first dataset and a second feature included in a second feature group corresponding to the second dataset; and a learning unit that performs a first learning process on an inference model using the dataset included in the first dataset, performs a second learning process on a first trained model acquired by the first learning process using the intermediate learning dataset, and performs a third learning process on a second trained model acquired by the second learning process using the dataset included in the second dataset.

2. The learning system according to claim 1, wherein the first imaging method is a method for imaging medical images, and the second imaging method is a method for imaging medical images that is different from the first imaging method, but uses the same body part as the first imaging method as the imaging target.

3. The learning system according to claim 2, wherein the first imaging method is either X-ray mammography or tomosynthesis, and the second imaging method is the other of X-ray mammography or tomosynthesis.

4. The learning system according to any one of claims 1 to 3, wherein the feature quantities are feature quantities obtained from an image.

5. The learning system according to any one of claims 1 to 4, wherein the second acquisition unit acquires the intermediate learning dataset based on the similarity or distance between the first feature and the second feature.

6. The learning system according to claim 5, wherein the second acquisition unit acquires the intermediate learning dataset based on at least one of the similarity or distance between the first feature and the second feature group, and the similarity or distance between the second feature and the first feature group.

7. The learning system according to claim 6, wherein the second acquisition unit acquires the intermediate learning dataset by at least one of the following processes: a first process of acquiring data having the first features similar to the second feature group from the first dataset, and a second process of acquiring data having the second features similar to the first feature group from the second dataset.

8. The learning system according to claim 5, wherein the second acquisition unit acquires a predetermined number of data from at least one of the first dataset and the second dataset as the intermediate learning dataset, in order of increasing similarity or decreasing distance.

9. The learning system according to claim 5, wherein the second acquisition unit acquires data from at least one of the first dataset and the second dataset as the intermediate learning dataset, wherein the similarity is greater than or equal to a threshold or the distance is less than or equal to a threshold.

10. The learning system according to any one of claims 1 to 4, wherein the second acquisition unit acquires the intermediate learning dataset based on the proportion of the first features included in the first feature group and the proportion of the second features included in the second feature group.

11. The learning system according to claim 10, wherein the second acquisition unit acquires the intermediate learning dataset from at least one of the first dataset and the second dataset such that the proportion of the third feature included in the third feature group corresponding to the intermediate learning dataset is between the proportion of the first feature included in the first feature group and the proportion of the second feature included in the second feature group.

12. The learning system according to any one of claims 1 to 11, wherein the second acquisition unit acquires a plurality of intermediate learning datasets, and the learning unit performs the second learning process step by step using the plurality of intermediate learning datasets.

13. The learning system according to any one of claims 1 to 12, wherein the second acquisition unit selects a dataset from which to acquire the intermediate learning dataset based on the breadth of the feature distribution or the amount of data of the first dataset and the second dataset.

14. The learning system according to any one of claims 1 to 13, wherein the second dataset is captured using a modality different from the modality used to capture the first dataset.

15. The learning system according to any one of claims 1 to 13, wherein the first dataset and the second dataset are captured using the same modality.

16. An inference device comprising: an acquisition unit for acquiring an image; and an inference unit for acquiring an inference result by inputting the image into a trained model, wherein the trained model is a trained model in which a first training process is performed on the inference model using a dataset included in a first dataset containing a plurality of data captured by a first shooting method; a second training process is performed on the first trained model acquired by the first training process using an intermediate training dataset; and a third training process is performed on the second trained model acquired by the second training process using a dataset included in a second dataset containing a plurality of data captured by a second shooting method different from the first shooting method, wherein the intermediate training dataset is acquired from at least one of the first dataset and the second dataset based on a first feature included in a first feature group corresponding to the first dataset and a second feature included in a second feature group corresponding to the second dataset.

17. A method for training an inference model, comprising: obtaining a first dataset containing multiple data captured using a first shooting method, and a second dataset containing multiple data captured using a second shooting method different from the first shooting method; obtaining an intermediate training dataset from at least one of the first dataset and the second dataset based on a first feature included in a first feature group corresponding to the first dataset and a second feature included in a second feature group corresponding to the second dataset; performing a first training process on an inference model using the dataset included in the first dataset; performing a second training process on a first trained model obtained by the first training process using the intermediate training dataset; and performing a third training process on a second trained model obtained by the second training process using the dataset included in the second dataset.

18. An inference method comprising: acquiring an image; and inputting the image into a trained model to obtain an inference result, wherein the trained model is a trained model which has been subjected to a first training process using a dataset included in a first dataset which includes a plurality of data captured by a first shooting method; a second training process which has been performed on the first trained model obtained by the first training process using an intermediate training dataset; and a third training process which has been performed on the second trained model obtained by the second training process using a dataset included in a second dataset which includes a plurality of data captured by a second shooting method different from the first shooting method, wherein the intermediate training dataset is obtained from at least one of the first dataset and the second dataset based on a first feature included in a first feature group corresponding to the first dataset and a second feature included in a second feature group corresponding to the second dataset.

19. A program that, when executed by a computer, causes the computer to perform the learning method for the inference model described in claim 17.

20. A program that, when executed by a computer, causes the computer to perform the inference method described in claim 18.

Images