Information processor, medical device, information processing method and program

Abstract

To improve discrimination performance for an image having similar aspect to an image of a different class of a learned model, when an image of a certain class of a data set is biased to an image having aspect less similar to the image of the different class, than an image having aspect more similar to the image of the different class, and reduce erroneous classification.SOLUTION: An information processor includes image acquisition means, correct answer class acquisition means, feature amount acquisition means, and selection means. The image acquisition means acquires a plurality of images. The correct answer class acquisition means acquires correct answer classes to which the respective images belong. The feature amount acquisition means acquires feature amounts of the respective images. The selection means selects an input image used in machine leaning from the image, on the basis of a relation between feature distribution relating to the feature amounts of the respective images belonging to the first correct answer class, and feature distribution relating to the feature amounts of the respective images belonging to the second correct answer class.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to an information processing device, a medical device, an information processing method, and a program.

[0002] Conventionally, there are known techniques for selecting an input image to be used in machine learning such as image classification from among a large number of images. For example, Patent Document 1 discloses a technique for clustering unlabeled data in a feature space during active learning, and uniformly selecting learning data from a predetermined number of clusters with low reliability. Patent Document 2 discloses a technique for adjusting the number of data so that the distribution of the features of the learning data becomes a predetermined distribution, and learning a model with the adjusted data set. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-167834 A [Patent Document 2] JP 2021-135583 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, by combining the techniques described in Patent Document 1 and Patent Document 2, it is possible to perform clustering based on features and select images from each cluster so that they have a predetermined distribution. However, if a model is trained on a data set in which images of a certain class are biased toward images that are more dissimilar than images that are similar in appearance to images of another class, the discrimination performance for images that are similar in appearance to images of another class may be reduced.

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the discrimination performance of a trained model for images that are similar in appearance to images of another class and reduce misclassification when images of a certain class in a dataset are biased toward images that are more dissimilar than images that are similar in appearance to images of another class. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0006] An information processing device according to an embodiment includes an image acquisition means, a correct class acquisition means, a feature acquisition means, and a selection means. The image acquisition means acquires a plurality of images. The correct class acquisition means acquires a correct class to which each of the images belongs. The feature acquisition means acquires features of each of the images. The selection means selects an input image to be used for machine learning from the images based on a relationship between a feature distribution regarding the feature amounts of each image belonging to a first correct class and a feature distribution regarding the feature amounts of each image belonging to a second correct class. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a tissue specimen image according to the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram for explaining the correct class according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram for explaining a pre-trained feature extraction model according to the first embodiment. [Diagram 5] FIG. 5 is a schematic diagram for explaining a pre-trained segmentation model according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of a tissue specimen image according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram showing an example of the positional relationship in the feature space between the normal class and the lesion class included in the tissue specimen image according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of processing executed by the image processing device according to the first embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of a configuration of an information processing system according to a modified example of the first embodiment. [Figure 10] FIG. 10 is a schematic diagram showing an example of processing of a small region image by an image processing device according to a modified example of the first embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram showing an example of the positional relationship in the feature space between the normal class and the lesion class included in the tissue specimen image according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of processing executed by the image processing device according to the second embodiment. [Figure 14] FIG. 14 is a block diagram showing an example of the configuration of an information processing system according to the third embodiment. [Figure 15] FIG. 15 is a schematic diagram showing an example of a learning data set in which images containing a mixture of adults and children are classified into males and females according to the third embodiment. [Figure 16] FIG. 16 is a table showing an example of how the number of images is determined according to the third embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of processing executed by the image processing device according to the third embodiment. [Figure 18] FIG. 18 is a schematic diagram illustrating an example of a learning data set in which images of children are classified as female according to the third embodiment. [Figure 19] FIG. 19 is a schematic diagram illustrating an example of a learning data set in which images of children are classified as female according to the third embodiment. [Figure 20]FIG. 20 is a diagram illustrating an example of the configuration of an information processing system according to the fourth embodiment. [Figure 21] FIG. 21 is a schematic diagram for explaining the contents learned by the image processing device according to the fourth embodiment. [Figure 22] FIG. 22 is a flowchart showing an example of processing executed by the image processing device according to the fourth embodiment. [Diagram 23] FIG. 23 is a block diagram showing an example of a configuration of an information processing system according to a modified example of the fourth embodiment. As shown in FIG. [Figure 24] FIG. 24 is a table showing an example of determining the number of images according to a modification of the fourth embodiment. [Diagram 25] FIG. 25 is a block diagram showing an example of the configuration of an information processing system according to the fifth embodiment. [Figure 26] FIG. 26 is a table showing an example of how the number of images is determined according to the fifth embodiment. [Figure 27] FIG. 27 is a flowchart showing an example of processing executed by the image processing device according to the fifth embodiment. [Figure 28] FIG. 28 is a block diagram showing an example of a configuration of an information processing system according to a modification of the fifth embodiment. As shown in FIG. [Figure 29] FIG. 29 is a block diagram showing an example of a configuration of an information processing system according to a modification of the sixth embodiment. As shown in FIG. [Diagram 30] FIG. 30 is a schematic diagram for explaining the contents of inference performed by the image processing device according to the sixth embodiment. [Diagram 31] FIG. 31 is a flowchart showing an example of processing executed by the image processing device according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an information processing device, a medical device, an information processing method, and a program according to the present embodiment will be described with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and duplicated descriptions will be omitted as appropriate.

[0009] (First embodiment) FIG. 1 is a block diagram showing an example of the configuration of an information processing system 100 according to the first embodiment. As shown in FIG. 1, the information processing system 100 includes a first storage device 10, a second storage device 20, and an image processing device 30. The first storage device 10, the second storage device 20, and the image processing device 30 are connected to each other so as to be able to communicate with each other via a network 40. Note that the configuration shown in FIG. 1 is an example, and the number of each device may be changed arbitrarily. Also, devices not shown in FIG. 1 may be connected to the network 40.

[0010] The first storage device 10 is an image storage device that stores sample images. The sample image is an image of the entire tissue slice that is the sample. The sample image is a digital image captured by an imaging device such as a digital camera. The sample image is also called a macro image. The first storage device 10 transmits the sample image of the pathological sample to the image processing device 30.

[0011] The first storage device 10 is also an image storage device that stores tissue images. A tissue image is an image of at least one slice cut out from a tissue piece that is a specimen, captured at a higher magnification than a specimen image. For example, a tissue image is a WSI (Whole Slide Imaging) image in which the entire slice or a part of the slice placed on a glass slide is digitalized with high accuracy. A tissue image may be a micrograph captured by a WSS (Whole Slide Scanner) or a digital microscope, or an image captured by other methods. A tissue image is also called a micro image.

[0012] The second storage device 20 is a correct class storage device that stores a correct class, which is a class to which each image belongs. There are a plurality of correct classes. The correct classes are stored in the second storage device 20, for example, with a first correct class as a normal class and a second correct class as a lesion class. The correct classes will be described in detail later.

[0013] The first storage device 10 and the second storage device 20 are, for example, a server device or a PC. Although the first storage device 10 and the second storage device 20 are illustrated as separate devices in Fig. 1, the first storage device 10 and the second storage device 20 may be configured as an integrated device. The first storage device 10, the second storage device 20, and the image processing device 30 may be configured as an integrated device.

[0014] The image processing device 30 is a computer device such as a personal computer, a server device, or a workstation. The image processing device 30 is an example of an information processing device. The image processing device 30 is also an example of a medical device. The image processing device 30 acquires a sample image from the first storage device 10. The image processing device 30 also acquires a correct class, which is a class to which each image belongs, from the second storage device 20. Furthermore, the image processing device 30 extracts a feature amount representing the characteristics of the image from the sample image acquired from the first storage device 10, and selects an input image to be used for machine learning based on the relationship with the class to which each image belongs.

[0015] Next, a detailed description will be given of the image processing device 30. For example, the image processing device 30 includes a NW (network) interface 310, an input interface 320, a display 330, a storage circuit 340, and a processing circuit 350.

[0016] The NW interface 310 is connected to the processing circuit 350, and controls the transmission and communication of various data between each device connected via the network 40. For example, the NW interface 310 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0017] The input interface 320 is connected to the processing circuitry 350, converts an input operation received from an operator into an electric signal, and outputs the electric signal to the processing circuitry 350. Specifically, the input interface 320 converts an input operation received from an operator into an electric signal, and outputs the electric signal to the processing circuitry 350. For example, the input interface 320 is realized by a trackball, a switch button, a mouse, a keyboard, a touchpad for performing an input operation by touching an operation surface, a touchscreen in which a display screen and a touchpad are integrated, a non-contact input circuit using an optical sensor, a voice input circuit, and the like.

[0018] In this specification, the input interface 320 is not limited to an interface having physical operation parts such as a mouse, a keyboard, etc. For example, an example of the input interface 320 also includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs the electrical signal to a control circuit.

[0019] The display 330 is connected to the processing circuit 350, and displays various information and various image data output from the processing circuit 350. For example, the display 330 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

[0020] The memory circuitry 340 is connected to the processing circuitry 350 and stores various data. The memory circuitry 340 also stores various programs that are read and executed by the processing circuitry 350 to realize various functions. For example, the memory circuitry 340 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, a hard disk, an optical disk, or the like.

[0021] The processing circuitry 350 controls the operation of the entire image processing device 30. The processing circuitry 350 has, for example, an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, and a probability determination function 355. In the embodiment, each processing function performed by the components of the image acquisition function 351, the correct class acquisition function 352, the feature acquisition function 353, the selection function 354, and the probability determination function 355 is stored in the storage circuitry 340 in the form of a program executable by a computer. The processing circuitry 350 is a processor that realizes a function corresponding to each program by reading and executing the program from the storage circuitry 340. In other words, the processing circuitry 350 in a state in which each program has been read has each function shown in the processing circuitry 350 in FIG. 1.

[0022] 1 is described as realizing the image acquisition function 351, correct class acquisition function 352, feature acquisition function 353, selection function 354, and probability determination function 355 by a single processor, but it is also possible to combine multiple independent processors to configure the processing circuit 350 and have each processor execute a program to realize the function. Also, in FIG. 1, it is described as a single storage circuit such as the storage circuit 340 storing a program corresponding to each processing function, but it is also possible to dispose multiple storage circuits in a distributed manner and have the processing circuit 350 read out the corresponding program from each storage circuit.

[0023] The term "processor" used in the above description refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). The processor realizes its function by reading and executing a program stored in the memory circuit 340. Note that instead of storing a program in the memory circuit 340, the program may be directly incorporated into the processor circuit. In this case, the processor realizes its function by reading and executing the program incorporated in the circuit.

[0024] The image acquisition function 351 acquires a plurality of images. The image acquisition function 351 is an example of an image acquisition means. Specifically, the image acquisition function 351 acquires a large number of small area images obtained by dividing a single tissue specimen image of a pathological sample into a grid pattern from the first storage device 10. Fig. 2 is a schematic diagram showing an example of a tissue specimen image. Fig. 2 shows a single tissue specimen image 11.

[0025] Moreover, the image acquisition function 351 may acquire a plurality of original images instead of acquiring a number of small area images from one original image (a tissue specimen image of a pathological specimen in this embodiment). Note that the original image is not limited to a tissue specimen image of a pathological specimen, but may be a two-dimensional image such as an ultrasound image, a fundus camera image, or a digital camera image, or may be a three-dimensional image such as a computed tomography (CT) image or a magnetic resonance imaging (MRI) image.

[0026] Returning to FIG. 1, the correct class acquisition function 352 acquires class information assigned to each of the acquired images as the correct class. The correct class acquisition function 352 is an example of a correct class acquisition means. For example, the correct class acquisition function 352 acquires, as the correct class, a "normal class" or a "lesion class" included in the class information assigned to each of the images acquired by the image acquisition function 351. The normal class is also referred to as the first correct class, and the lesion class is also referred to as the second correct class.

[0027] Here, the correct class will be explained with reference to Fig. 3. Fig. 3 is a schematic diagram for explaining the correct class. Fig. 3 shows small region images 12 acquired from a tissue specimen image, a classifier 13, and classes 14. The small region images 12 are images obtained by dividing one tissue specimen image into a grid pattern. For example, the correct class acquisition function 352 acquires a large number of small region images 12 from one tissue specimen image, and acquires class information indicating whether each of the small region images 12 is a normal class or a lesion class assigned by a doctor.

[0028] Return to FIG. 1. The feature acquisition function 353 acquires features expressing the features of each of the acquired multiple images. The feature acquisition function 353 is an example of a feature acquisition means. The feature is, for example, an array of RGB values ​​of each pixel, an output vector of an intermediate layer of a pre-trained feature extraction model, or an output of a pre-trained segmentation model. Each pixel is a voxel if it is a three-dimensional image. Also, the RGB value is a luminance value if it is a monochrome image. For example, the feature acquisition function 353 acquires the reduced dimensionality of the RGB values ​​of each image as the feature.

[0029] The output vector of the intermediate layer of the pre-trained feature extraction model will be judged for its class-likeness that was set when the model was trained, but it is expected that pixels with similar features will have similar outputs, and pixels with different features will have different outputs, so the pre-training does not have to be targeted at the target task or target data.

[0030] Here, an example of using a pre-trained feature extraction model will be described with reference to Fig. 4. Fig. 4 is a schematic diagram for explaining a pre-trained feature extraction model. Fig. 4 shows a plurality of images 15 of small regions acquired from a tissue specimen image, and feature amounts 16 of the images of each small region. For example, in the case of images 15 shown in Fig. 4, it is assumed that similar feature amounts are extracted between images of lymph node portions and between images of fat portions, and different feature amounts are extracted between images of lymph node portions and images of fat portions.

[0031] The output of the pre-trained segmentation model can be used as a feature when extracting a small region of an image, such as a tissue specimen image of a pathology sample. For example, the array of output values ​​for each pixel can be used as a feature.

[0032] Here, an example of using a pre-trained segmentation model will be described with reference to FIG. 5. FIG. 5 is a schematic diagram for explaining a pre-trained segmentation model. FIG. 5 shows a pixel image 111 obtained by classifying a tissue specimen image, and a partial pixel image 1111 included in the pixel image 111. It is also assumed that the pixel image 111 is classified into class 1 and class 2. For example, the proportion of pixels classified into class 2 in the partial image can be used as a feature amount. The partial pixel image 1111 shown in FIG. 5 has many pixels of class 2.

[0033] Therefore, if a partial image including a lesion has a high proportion of pixels in class 2, the partial pixel image 1111 is more likely to be selected when selecting an image by the selection function 354 described later, since the partial pixel image 1111 has similar features to the partial image including the lesion than other partial images with a low proportion of pixels classified into class 2. Furthermore, when grouping by the grouping function 357 described later, the partial pixel image 1111 and other partial images with a low proportion of pixels classified into class 2 are likely to be classified into different groups. Note that the pre-learning does not have to be targeted at the target task or data.

[0034] Return to FIG. 1. The selection function 354 selects an input image to be used for machine learning from the images based on the relationship between the feature distribution of the feature amount of each image belonging to the first correct class and the feature distribution of the feature amount of each image belonging to the second correct class. The selection function 354 is an example of a selection means. Specifically, the selection function 354 selects, among the images belonging to the first correct class, images distributed close to the second correct class in preference to images distributed far from the second correct class. For example, the selection function 354 selects all images of the lesion class and randomly selects the normal class based on a set probability.

[0035] Here, a method for selecting an input image will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a schematic diagram showing an example of a tissue specimen image. Fig. 7 is a schematic diagram showing an example of a positional relationship in a feature space between a normal class and a lesion class contained in the tissue specimen image.

[0036] 6 shows a normal class 1121 of the lymph node portion, a lesion class 1122, and a normal class 1123 of the fat portion, which are included in the tissue specimen image 112. Also, Fig. 7 shows a normal class 141 of the fat portion, a normal class 142 of the lymph node portion, a lesion class 143, a center of gravity 144 of the lesion class, a distance 145 between the normal class 142 of the lymph node portion and the center of gravity 144 of the lesion class, and a distance 146 between the normal class 141 of the fat portion and the center of gravity 144 of the lesion class.

[0037] As shown in Fig. 6, an image belonging to normal class 1121 of the lymph node portion has similar features to an image belonging to lesion class 1122. Therefore, as shown in Fig. 7, normal class 142 of the lymph node portion has a small distance 145 from center of gravity 144 of the lesion class. On the other hand, as shown in Fig. 6, an image belonging to normal class 1123 of the fat portion has dissimilar features to an image belonging to lesion class 1122. Therefore, as shown in Fig. 7, normal class 141 of the fat portion has a large distance 146 from center of gravity 144 of the lesion class.

[0038] For example, the selection function 354 calculates the center of gravity (center of gravity of the lesion class) of the feature distribution of the images belonging to the second correct class (lesion class), and randomly selects the images belonging to the first correct class (normal class) by setting the probability of selection (probability determination means) to be inversely proportional to the distance from the center of gravity. This makes it possible to prevent bias toward images that are not similar to the second correct class and to reduce misclassification of images similar to the second correct class.

[0039] Returning to FIG. 1, the probability determination function 355 determines the probability that each image is selected and the probability that an image is selected from each group. The probability determination function 355 is an example of a probability determination means. Specifically, the probability determination function 355 determines the probability that each image is selected based on the relationship between the feature distribution regarding the feature amount of each image belonging to the first correct class and the feature distribution regarding the feature amount of each image belonging to the second correct class.

[0040] Furthermore, the above-mentioned selection function 354 selects an input image based on the probability determined by the probability determination function 355. For example, the selection function 354 preferentially selects images distributed near the second correct class among images belonging to the first correct class over images distributed far from the second correct class. Furthermore, in the above-mentioned selection function 354, an example has been given in which the probability of an image belonging to the first correct class being selected is set so as to be inversely proportional to the distance from the center of gravity of the feature distribution of an image belonging to the second correct class, but any method for calculating the probability may be used as long as the probability of a close image is higher than the probability of a distant image, and the center of gravity of the feature distribution need not be used as a criterion for calculating the distance.

[0041] For example, the method of setting the probability may use a value obtained by inputting distance into a linear function with a negative slope, or in the case of groups, the similarity of the feature amounts of images belonging to each group. Also, the method of setting the probability does not have to be inversely proportional to the distance, as long as a higher probability is given to a closer image than to a more distant image. Furthermore, the method of setting the probability may use any function in which the probability decreases as the distance increases (probability is monotonically non-increasing with respect to distance).

[0042] For example, the criterion for calculating the distance may be a vector of the median of each dimension of the feature, a representative point selected from a high-density part in a class group, etc. In addition, the criterion for calculating the distance may be to select a sample close to the center of gravity of the second correct class as the representative point, and use the distance from the representative point.

[0043] Furthermore, the distance is calculated based on the minimum distance from multiple representative points of the second correct class. The distance is calculated based on the minimum distance from all images of the second correct class (= distance to the nearest image). The distance is calculated based on the minimum distance from the center of gravity of each group, grouping the second correct class based on features, etc. Each of the above-mentioned minimum values ​​may be an average value.

[0044] The criterion for calculating the distance between point clouds and points may be any method for measuring the distance to the point clouds, or between point clouds (described later), the criterion for calculating the distance may be any method for measuring the distance between the point clouds.

[0045] Next, a description will be given of a process executed by the image processing device 30 according to the first embodiment. Fig. 8 is a flowchart showing an example of a process executed by the image processing device 30 according to the first embodiment.

[0046] First, the image acquisition function 351 acquires a small region image of one tissue specimen image of a pathological sample from the first storage device 10 (step S11). Next, the correct class acquisition function 352 acquires class information indicating whether the acquired image belongs to a normal class or a lesion class (step S12). Next, the feature acquisition function 353 acquires the reduced dimensionality of the RGB values ​​of each image as a feature (step S13).

[0047] Next, the probability determination function 355 calculates the center of gravity of the feature of the lesion class, and determines the probability that an image of the normal class will be selected so as to be inversely proportional to the distance between the feature and the center of gravity of the lesion class (step S14). Next, the selection function 354 selects all images of the lesion class, and randomly selects images of the normal class based on the set probability (step S15). When step S15 is completed, this process ends.

[0048] As described above, the image processing device 30 according to the first embodiment acquires a plurality of images, acquires a correct class which is a class to which each of the plurality of images belongs from among the plurality of images, acquires feature amounts of each of the plurality of images, and selects an input image to be used for machine learning from the images based on the relationship between the feature distribution regarding the feature amounts of each image belonging to the first correct class and the feature distribution regarding the feature amounts of each image belonging to the second correct class.

[0049] As a result, for example, when images of a certain class in a dataset are biased toward images that are more dissimilar in appearance to images of another class than images that are similar in appearance to images of another class, the image processing device 30 can improve the discrimination performance of the learned model for images that are similar in appearance to images of another class and reduce misclassification.

[0050] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each device. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. The following mainly describes the differences from the above-described embodiment, and the same reference numerals will be given to the points that are common to the contents already described, and detailed description will be omitted. The other embodiments described below may be implemented individually or in appropriate combination.

[0051] (Modification of the first embodiment) 9 is a block diagram showing an example of the configuration of an information processing system 100 according to a modified example of the first embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a probability determination function 355, and a data expansion function 356.

[0052] The data expansion function 356 performs data expansion of the acquired image, such as rotation, inversion, color conversion, etc. The data expansion function 356 is an example of a data expansion means. Here, the data expansion process will be described with reference to FIG. 10. FIG. 10 is a schematic diagram showing an example of processing of a small region image by the image processing device 30 according to the modified example of the first embodiment. FIG. 10 shows a small region image 121 that has been rotated 90° to the right and data expanded by the data expansion function 356 with respect to the small region image 12 shown in FIG. 3. The data expansion function 356 may perform data expansion process on the selected image, for example, in step S15 shown in FIG. 8.

[0053] For example, if there is a group with a small number of images and the number of images in other groups is determined accordingly, the variation of learning images may be reduced and generalization performance may decline. Therefore, it is effective for the data expansion function 356 to inflate the number of images by performing data expansion on images selected from a group with a small number of images, and to determine the number of images in other groups accordingly.

[0054] As a modified example of the image processing device 30 of the first embodiment, for example, in step S13 shown in Fig. 8, the feature acquisition function 353 may use the output of the intermediate layer when an image is input to a pre-trained feature extraction model. The class-likeness set during model training is judged, but since it is assumed that pixels with similar features will have similar outputs and pixels with different features will have different outputs, the pre-training does not have to be performed for the target task or target data.

[0055] Furthermore, in step S13 shown in Fig. 8, the feature acquisition function 353 may use an output corresponding to the position of each small region image when the tissue specimen image is input to the pre-trained segmentation model. As with the above-mentioned feature extraction model, the pre-training does not have to be performed for the target task or target data. In that case, an array of output values ​​for each pixel of the small region image or an array of classes into which each pixel is classified is used as a feature.

[0056] Second embodiment 11 is a block diagram showing an example of the configuration of an information processing system 100 according to the second embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a probability determination function 355, and a grouping function 357.

[0057] The grouping function 357 classifies images belonging to each correct answer class into groups based on the feature amount. The grouping function 357 is an example of a grouping means. Specifically, the grouping function 357 classifies images belonging to a first correct answer class into a plurality of groups based on the feature amount. For example, the grouping function 357 classifies images into at least two groups for at least one correct answer class, although the number of groups may differ for each correct answer class. For example, the normal class is classified into two groups, a lymph node area and a fat area, and the lesion class is classified into the same, single group for all images.

[0058] Here, the grouping method will be described with reference to FIG. 12. FIG. 12 is a schematic diagram showing an example of the positional relationship in feature space between a normal class and a lesion class contained in a tissue specimen image. FIG. 12 shows a threshold boundary 147 in the schematic diagram of FIG. 7. The grouping method, for example, involves clustering based on features for each class. The grouping method also involves calculating the distance from the center of gravity of the feature distribution of images of different classes, and dividing into a group below the threshold (inside the threshold boundary 147 shown in FIG. 12) and a group above the threshold (outside the threshold boundary 147 shown in FIG. 12).

[0059] As a result, images of lymph nodes are mainly classified into the group below the threshold, and images of fat are mainly classified into the group above the threshold. Therefore, if the probability of being selected from each group is set to be inversely proportional to the distance between the center of gravity of each group and the center of gravity of the lesion class, the group below the threshold is more likely to be selected. Furthermore, when the output of a pre-trained segmentation model is used as a feature, the grouping method is based on the proportion of classes into which each pixel is classified.

[0060] Then, the selection function 354 selects an input image from each of the groups based on the relationship between the feature distribution regarding the feature amounts of each image belonging to the second correct class and the feature distribution regarding the feature amounts of images belonging to each of the groups grouped by the grouping function 357. For example, the selection function 354 preferentially selects images belonging to groups distributed near the second correct class among the multiple groups over images belonging to groups distributed far from the second correct class.

[0061] Also, for example, the probability determination function 355 determines the probability of selecting an input image from each of the groups based on the relationship between the feature distribution for the features of each image belonging to the second correct class and the feature distribution for the features of images belonging to each of the groups grouped by the grouping function 357.

[0062] More specifically, the probability determination function 355 determines the probability that an image is selected from each group of the normal class so as to be inversely proportional to the distance between the center of gravity of that group and the center of gravity of the lesion class. Then, the selection function 354 selects an input image from each of the groups based on the probability determined by the probability determination function 355. More specifically, the selection function 354 selects all images of the lesion class and randomly selects images of the normal class from each group based on the set probability.

[0063] Next, a process executed by the image processing device 30 according to the second embodiment will be described. Fig. 13 is a flowchart showing an example of a process executed by the image processing device 30 according to the second embodiment. Also, a description of the process common to the process executed by the image processing device 30 according to the first embodiment shown in Fig. 8 will be omitted.

[0064] In step S21, the grouping function 357 calculates the center of gravity of the feature of the lesion class, and classifies the normal class into a group in which the distance between the feature and the center of gravity of the lesion class is less than a threshold and a group in which the distance is greater than or equal to the threshold (step S21).

[0065] In step S22, the probability determination function 355 determines the probability that an image is selected from each group of the normal class so as to be inversely proportional to the distance between the center of gravity of that group and the center of gravity of the lesion class (step S22).

[0066] In step S23, the selection function 354 selects all images in the lesion class, and randomly selects images in the normal class from each group based on a set probability (step S23). When the image processing device 30 ends step S23, this process ends.

[0067] As described above, the image processing device 30 according to the second embodiment classifies images belonging to the first correct class into a plurality of groups based on their features, and selects an input image from each of the groups based on the relationship between the feature distribution regarding the features of each image belonging to the second correct class and the feature distribution regarding the features of images belonging to each of the groups.

[0068] Thereby, for example, by performing grouping, the image processing device 30 can increase the probability that an image in the first correct class group that is distributed closer to an image belonging to the second correct class will be selected.

[0069] (Modification of the second embodiment) As a modified example of the image processing device 30 of the second embodiment, for example, in step S21 shown in FIG. 13, the grouping function 357 may classify the normal class into groups by clustering using feature amounts.

[0070] As a modified example of the image processing device 30 of the second embodiment, for example, in step S13 shown in Fig. 13, the feature acquisition function 353 may use an output corresponding to the position of each small area image when the tissue specimen image is input to the pre-trained segmentation model. In this case, in step S24, the grouping function 357 may perform grouping by classifying each pixel of the small area image into the class group with the highest classification rate.

[0071] Third embodiment FIG. 14 is a block diagram showing an example of the configuration of the information processing system 100 according to the third embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a grouping function 357, and an image number determination function 358. The first storage device 10 in the third embodiment stores a plurality of images of a mixture of adults and children. The correct class stored in the second storage device 20 in the third embodiment is a male / female class, and for example, the first correct class is stored as a male class and the second correct class is stored as a female class. The first correct class may be a female class and the second correct class may be a male class.

[0072] The image acquisition function 351 acquires multiple images of a mixture of adults and children. The correct class acquisition function 352 acquires information on whether each image belongs to a male class or a female class. The feature acquisition function 353 inputs images into a model pre-trained with an open dataset and acquires the output of the intermediate layer as features. The grouping function 357 clusters images of each class based on the features.

[0073] The number of images determination function 358 determines the number of images to be selected from each group based on the relationship between the feature distributions regarding the feature amounts of the images belonging to each of the groups. The number of images determination function 358 is an example of an image number determination means. Specifically, the number of images determination function 358 determines the number of images of the input images to be selected from each of the groups based on the relationship between the feature distributions regarding the feature amounts of the images belonging to the second correct answer class and the feature distributions regarding the feature amounts of the images belonging to each of the groups. For example, the number of images determination function 358 determines the number of images to be selected from the clusters distributed near the other child cluster so that the number of images to be selected from the clusters distributed farther away is greater than the number of images to be selected from the clusters distributed near the other child cluster.

[0074] Here, a method for determining the number of images will be described with reference to FIG. 15 and FIG. 16. FIG. 15 is a schematic diagram showing an example of a learning data set in which images of adults and children are classified into males and females. FIG. 15 shows an adult male image group GR1, an adult female image group GR2, a male child image group GR3, a female child image group GR4, a distance 151 indicating the distance between the adult female image group GR2 and the male child image group GR3, a distance 152 indicating the distance between the adult male image group GR1 and the female child image group GR4, and a distance 153 indicating the distance between the male child image group GR3 and the female child image group GR4. FIG. 16 is a table showing an example of determining the number of images. FIG. 16 shows a table 154 that was the reference before the number of images was determined, and a table 155 after the number of images was determined.

[0075] For example, when determining the number of images to select from the adult female image group GR2 and the female child image group GR4 based on the relationship between the feature distributions regarding the features of images belonging to the male child image group GR3, the image number determination function 358 determines the number of images to select from the female child image group GR4 ("similar group", "proximal group") which is distributed near the male child image group GR3, and the number of images to select from the adult female image group GR2 ("dissimilar group", "non-proximal group") which is distributed farther away, so that more images are selected from the female child image group GR4 ("similar group") than from the adult female image group GR2 ("dissimilar group").

[0076] This makes it possible to prevent bias toward images distributed far away in the image group GR3 of male children. Similarly, the number of images to be selected from the image group GR1 of adult men and the number of images to be selected from the image group GR3 of male children are also determined based on the relationship between the feature distribution regarding the feature amount of images belonging to the image group GR4 of female children.

[0077] The method of determining the number of images is, for example, to set the number of images selected from each group to a predetermined number. The method of determining the number of images is to set the ratio of the number of images in the group distributed close to the second correct class to the number of images in the group distributed far away to a predetermined ratio (for example, 2:1) (see table 155 in FIG. 16). In other words, the number of images determination function 358 determines the number of images so that the number of input images selected from the group distributed close to the second correct class among the multiple groups is equal to or greater than the number of input images from the group distributed far away from the second correct class. Also, the method of determining the number of images is to determine the number of images so that it is inversely proportional to the distance (defined between the groups).

[0078] The number of images may be determined based on the number of images in each group in addition to the relationship between the feature distributions. That is, the number of images determination function 358 determines the number of images based on the number of images in each group. For example, the number of first groups may be the same as the number of second groups, or the sum of the number of first and second groups may be the same as the number of third groups.

[0079] As a result, since images of children have features more similar to images of children in the other class than to images of adults in the same class, it is expected that if images of one class are biased toward adult images, images of children will be more likely to be misclassified into the other class. By selecting more clusters closer to the other children's cluster, it is possible to reduce the bias toward images of adults.

[0080] A selection function 354 selects a determined number of input images from each of the groups.

[0081] Next, a process executed by the image processing device 30 according to the third embodiment will be described below. Fig. 17 is a flowchart showing an example of a process executed by the image processing device 30 according to the third embodiment.

[0082] First, the image acquisition function 351 acquires a plurality of images of a mixture of adults and children (step S31). Next, the correct class acquisition function 352 acquires information on whether each image belongs to a male class or a female class (step S32). Next, the feature acquisition function 353 inputs the image into a model pre-trained with an open dataset, and acquires the output of the intermediate layer as a feature (step S33).

[0083] Next, the grouping function 357 performs clustering on the images of each class based on the feature amount (step S34). Next, the image number determination function 358 determines the number of images selected from the clusters distributed near the other child cluster so that the number of images selected from the clusters distributed farther away is greater (step S35). Next, the selection function 354 selects the determined number of images from each group (step S36). When the process of step S36 ends, this process ends.

[0084] As described above, the image processing device 30 according to the third embodiment determines the number of input images to be selected from each group based on the relationship between the feature distribution regarding the feature amounts of each image belonging to the second correct class and the feature distribution regarding the feature amounts of images belonging to each group. This allows the image processing device 30 to determine the number of images to be selected such that, for example, the number of images to be selected is greater for the images of the first correct class group that are distributed closer to the images belonging to the second correct class.

[0085] (Modification of the third embodiment) As a modified example of the image processing device 30 of the third embodiment, for example, in step S36, the selection function 354 may select images evenly for each distance from another group. A method in which the selection function 354 selects images evenly will be described with reference to Figs. 18 and 19. Figs. 18 and 19 are schematic diagrams showing an example of a learning dataset in which images of children are classified as female.

[0086] Fig. 18 shows a female child image group GR41, which indicates a state that may occur when the selection function 354 places importance on the distance from another group for the female child image group GR4 shown in Fig. 15. Fig. 19 shows a female child image group GR42, which indicates a state that may occur when the selection function 354 selects images equally for each distance for the female child image group GR4 shown in Fig. 15.

[0087] For example, if the selection function 354 preferentially selects images that are close to another group as shown in Fig. 18, the variation of images within the group may decrease, and generalization performance may decrease. Therefore, as a modification of the modification of the third embodiment, the selection function 354 selects images evenly for each distance from another group, or selects images with a large variance, as shown in Fig. 19. This makes it possible to suppress bias toward images that are close to another group.

[0088] As a modified example of the image processing device 30 of the third embodiment, for example, in step S36, the selection function 354 may select an image based on the result of reducing the dimension of the feature amount of the images in the group. For example, when the images in the group are distributed in a Swiss roll shape (distributed in a spiral shape in three-dimensional space), even if the images are selected equally based on the distance from another group, some images will be biased, and even if a selection is made to increase the variance, images distributed in the center are difficult to select. Therefore, the selection function 354 may perform a balanced selection by reducing the dimension from three dimensions to two dimensions based on a known dimensionality reduction method.

[0089] (Fourth embodiment) Fig. 20 is a diagram showing an example of the configuration of an information processing system 100 according to the fourth embodiment. As shown in Fig. 20, the information processing system 100 according to the fourth embodiment includes a first storage device 10, a second storage device 20, an image processing device 30, a third storage device 50, and a fourth storage device 60. The first storage device 10, the second storage device 20, the image processing device 30, the third storage device 50, and the fourth storage device 60 are connected to each other via a network 40 so as to be able to communicate with each other.

[0090] The first storage device 10 in the fourth embodiment stores a plurality of images of a mixture of adults and children, similar to the first storage device 10 in the third embodiment. Also, the second storage device 20 in the fourth embodiment stores the correct class stored in the second storage device 20 as a male / female class, similar to the second storage device 20 in the third embodiment, and stores, for example, the first correct class as a male class and the second correct class as a female class. Note that the first correct class may be the female class and the second correct class may be the male class.

[0091] The third storage device 50 is a dataset storage device that stores a dataset. The third storage device 50 stores a file output by a dataset function of the processing circuit 350, which will be described later. The fourth storage device 60 is a trained model storage device that stores trained models. The fourth storage device 60 stores trained models trained by a learning function 360 of the processing circuit 350, which will be described later.

[0092] The processing circuit 350 of the image processing device 30 has an image acquisition function 351 , a correct class acquisition function 352 , a feature acquisition function 353 , a selection function 354 , a grouping function 357 , an image number determination function 358 , a dataset construction function 359 and a learning function 360 .

[0093] The dataset construction function 359 constructs a dataset using the selected image. The dataset construction function 359 is an example of a dataset construction means. Specifically, the dataset construction function 359 constructs a dataset using the image selected by the selection function 354. Then, the dataset construction function 359 outputs a file that describes the path of the selected image and information on the class to which the image belongs.

[0094] The learning function 360 learns a machine learning model using a dataset configured from the selected images. The learning function 360 is an example of a learning unit. Specifically, the learning function 360 learns a machine learning model using a dataset configured based on the input images selected by the selection function 354. For example, the learning function 360 learns a machine learning model using a file output by the dataset configuration function 359.

[0095] Here, the content of constructing a data set using selected images and learning a model by the image processing device 30 will be described with reference to Fig. 21. Fig. 21 is a schematic diagram for explaining the content of learning by the image processing device 30 according to the fourth embodiment.

[0096] The image acquisition function 351 of the image processing device 30 acquires a plurality of images 161 that include a mixture of adults and children (image selection process 162 shown in FIG. 21). Next, the dataset construction function 359 constructs a dataset using the selected images (dataset construction process 163 shown in FIG. 21). The dataset construction function 359 then outputs a file that describes the path of the selected image and information on the class to which the image belongs. For example, the table 164 showing the list of files shown in FIG. 21 shows combinations of image paths and correct classes, such as "Image1" and "male," "Image2" and "female," "Image3" and "male," and "Image4" and "female."

[0097] The learning function 360 learns the machine learning model using the file output by the dataset configuration function 359. For example, as shown in FIG. 21, the learning function 360 reads an image and a correct answer class ("Image1 / male" 165 shown in FIG. 21) selected from the file output by the dataset configuration function 359, performs a learning process 166 on the AI ​​for classifying male and female, and outputs a model 167.

[0098] Next, a process executed by the image processing device 30 according to the fourth embodiment will be described. Fig. 22 is a flowchart showing an example of a process executed by the image processing device 30 according to the fourth embodiment. Also, a description of the process common to the process executed by the image processing device 30 according to the third embodiment shown in Fig. 17 will be omitted.

[0099] In step S37, the data set construction function 359 outputs a file that describes the path of the selected image and information on the class to which the image belongs (step S37). In step S38, the learning function 360 reads the selected image and the correct class from the file output by the data set construction function 359, and learns the male / female classification AI (step S38). When the processing circuit 350 ends step S38, this process ends.

[0100] As described above, the image processing device 30 according to the fourth embodiment trains a machine learning model using a data set constructed based on selected input images. As a result, for example, when images of a certain class in a data set are biased toward images that are more dissimilar than images that are similar in appearance to images of another class, the image processing device 30 can improve the discrimination performance of the trained model for images that are similar in appearance to images of another class, and reduce misclassification.

[0101] (Modification of the fourth embodiment) 23 is a block diagram showing an example of the configuration of an information processing system 100 according to a modified example of the fourth embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a data expansion function 356, a grouping function 357, an image number determination function 358, a data set configuration function 359, a learning function 360, and an inference function 361.

[0102] The inference function 361 infers a class to which a newly input image belongs, using the machine learning model learned by the learning function 360. The inference function 361 is an example of an inference means. For example, the inference function 361 may perform inference on the input image, using the model learned by the learning function 360, in step S38 shown in FIG.

[0103] Also, for example, the data set construction function 359 may construct a data set by regarding the images selected from each group as belonging to different classes in step S37 shown in Fig. 22. For example, the different classes are four classes: 1st male, 2nd male, 1st female, and 2nd female. In this case, the learning function 360 learns a model for classifying images into the determined classes in step S38 shown in Fig. 22.

[0104] In addition, the inference function 361 may output the class inferred by the model as is, or may output a result that if the person is classified into either the first male or the second male, the person belongs to the male class, or may output a result that if the person is classified into one of the same original class, the person belongs to the original class.

[0105] Furthermore, for example, in step S36 shown in Fig. 22, the data expansion function 356 performs image data expansion such as rotation, inversion, color conversion, etc. on the number of images determined from each group selected by the selection function 354. Then, in step S37 shown in Fig. 22, the data set configuration function 359 may output a file describing the paths of the selected images and the data-expanded images and information on the class to which the images belong.

[0106] Also, for example, in step S38 shown in FIG. 22, the learning function 360 may train a machine learning model using input images so that the total number of input images selected from each group by the selection function 354 and images data-extended by the data expansion function 356 is the number of images determined in step S36 shown in FIG. 22.

[0107] In this case, the number of images determination function 358 may determine the number of images in each group to exceed the number of images that originally belong to the group. Here, the method of determining the number of images will be described with reference to Fig. 24. Fig. 24 is a table showing an example of determining the number of images. Fig. 24 shows table 171, which was the reference before the number of images was determined, and table 172 after the number of images was determined.

[0108] If the number of images determination function 358 determines the number of images for other groups based on the number of images for a group with a smaller number of images, the variation of training images may decrease, and generalization performance may decline (Table 171 shown in FIG. 24). Therefore, the number of images determination function 358 can prevent the variation from decreasing by setting the number of images to include data-augmented images (Table 172 shown in FIG. 24).

[0109] Additionally, the selection function 354 may select fewer images from each group than the determined number of images. Additionally, the dataset construction function 359 may construct a dataset based on the selected images and the data-augmented images.

[0110] Fifth embodiment 25 is a block diagram showing an example of the configuration of an information processing system 100 according to the fifth embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a grouping function 357, an image number determination function 358, and a bias determination function 362.

[0111] The first storage device 10 in the fifth embodiment stores a plurality of images of a mixture of adults and children, similar to the first storage device 10 in the third embodiment. Also, the second storage device 20 in the fifth embodiment stores the correct classes stored in the second storage device 20 as classes of males and females, similar to the second storage device 20 in the third embodiment, and stores, for example, the first correct class as a male class and the second correct class as a female class. Note that the first correct class may be the female class and the second correct class may be the male class.

[0112] The bias determination function 362 determines whether the feature distribution of images belonging to the first correct answer class is biased toward either images distributed near the group of the second correct answer class or images distributed far away. The bias determination function 362 is an example of a bias determination means. Specifically, if the bias determination function 362 determines that there is bias, the selection function 354 performs an image selection process. Also, if the bias determination function 362 determines that there is no bias, the selection function 354 selects all images belonging to the first correct answer class as input images to be used in machine learning.

[0113] For example, bias determination function 362 calculates the ratio of images distributed near the second correct class group to images distributed far away, and determines bias based on whether the ratio is less than a threshold. Next, image number determination function 358 determines the number of images to select from each group only when the images of the first correct class are biased toward either images distributed near the second class group or images distributed far away. Then, selection function 354 selects the determined number of images from each group.

[0114] Here, the content of the selection function 354 selecting a determined number of images from each group based on the result of the determination by the bias determination function 362 will be described with reference to Fig. 26. Fig. 26 is a table showing an example of the determined number of images.

[0115] For example, the bias determination function 362 determines whether the number of images in a group distributed near the other child cluster of one correct answer class is less than half the number of images in a group distributed farther away. Here, the group distributed near the other child cluster of one correct answer class is a child female class, and the group distributed farther away is an adult female class. In the case of table 156 shown in FIG. 26, the number of images in the child female class group and the number of images in the adult female class group are both 5,000.

[0116] In the case of table 156 shown in Fig. 26, bias determination function 362 determines that there is no bias because the number of images in a group distributed near one child cluster of the other correct answer class is not less than half the number of images in a group distributed farther away (the number of images is the same). Furthermore, number of images determination function 358 selects all the numbers of images from each group and determines the number of images shown in table 156. Then, selection function 354 selects the determined number of images from each group for the number of images shown in table 156 shown in Fig. 26.

[0117] Furthermore, when the bias determination function 362 determines that the feature distribution of images belonging to the first correct class is biased toward either images distributed close to the group of the second correct class or images distributed far away, the number of images determined by the image number determination function 358 and the selection of images selected by the selection function 354 are processed in the same manner as in the third embodiment described above. As a result, the image processing device 30 can prevent unnecessary processing from being performed when there is no bias by performing image selection processing by the selection function 354 only when it is determined that there is bias.

[0118] Next, a process executed by the image processing device 30 according to the fifth embodiment will be described. Fig. 27 is a flowchart showing an example of a process executed by the image processing device 30 according to the fifth embodiment. Also, a description of the process common to the process executed by the image processing device 30 according to the third embodiment shown in Fig. 17 will be omitted.

[0119] First, in step S41, the bias determination function 362 determines whether the number of images in the cluster distributed near the other child cluster is less than half the number of images in the cluster distributed far away (step S41). If the bias determination function 362 determines that the number of images in the cluster distributed near the other child cluster is less than half the number of images in the cluster distributed far away (step S41: Yes), the processing circuit 350 proceeds to step S35.

[0120] On the other hand, if the bias determination function 362 determines that the number of images in the cluster distributed near the other child cluster is more than half the number of images in the cluster distributed farther away, and is not biased (step S41: No), the processing circuit 350 proceeds to step S42. In step S42, the selection function 354 selects all images as input images to be used for machine learning (step S42). When the processing circuit 350 ends step S42, this process ends.

[0121] As described above, the image processing device 30 according to the fifth embodiment determines whether the feature distribution of images belonging to the first correct class is biased toward either images distributed near the group of the second correct class or images distributed far away, and performs an image selection process if it is determined that the feature distribution is biased, and selects all images belonging to the first correct class as input images to be used in machine learning if it is determined that the feature distribution is not biased. As a result, for example, when images of a certain class in a dataset are biased toward images that are dissimilar to images of another class rather than images that are similar in appearance to images of another class, the image processing device 30 can improve the discrimination performance of the learned model for images that are similar in appearance to images of another class and reduce misclassification.

[0122] (Modification of the fifth embodiment) 28 is a block diagram showing an example of a configuration of an information processing system 100 according to a modified example of the fifth embodiment. The processing circuit 350 of the image processing device 30 has an image acquisition function 351, a correct class acquisition function 352, a feature acquisition function 353, a selection function 354, a grouping function 357, an image number determination function 358, a bias determination function 362, and an instruction acquisition function 363.

[0123] The instruction acquisition function 363 acquires an instruction as to whether or not to perform image selection processing by the selection function 354. The instruction acquisition function 363 is an example of an instruction acquisition unit. Specifically, when the bias determination function 362 determines that there is bias and the instruction acquisition function 363 acquires an instruction to select an image, the selection function 354 performs image selection processing. Also, when the bias determination function 362 determines that there is no bias or when the instruction acquisition function 363 does not acquire an instruction to select an image, the selection function 354 selects all images belonging to the first correct class as input images to be used in machine learning.

[0124] For example, when it is determined that there is a bias in step S41 shown in Fig. 26 (step S41: Yes), the instruction acquisition function 363 requests an instruction from the user as to whether or not to perform image selection processing. When the instruction acquisition function 363 acquires an instruction to perform selection processing, the instruction acquisition function 363 executes image selection processing. Moreover, when the instruction acquisition function 363 acquires an instruction not to perform selection processing, the instruction acquisition function 363 may select all images as input images to be used in machine learning.

[0125] Sixth embodiment 29 is a block diagram showing an example of a configuration of an information processing system 100 according to a modification of the sixth embodiment. The processing circuitry 350 of the image processing device 30 has an inference function 361.

[0126] The first storage device 10 in the sixth embodiment stores a plurality of images of a mixture of adults and children, similar to the first storage device 10 in the third embodiment. Also, the second storage device 20 in the sixth embodiment stores the correct classes stored in the second storage device 20 as classes of males and females, similar to the second storage device 20 in the third embodiment, and stores, for example, the first correct class as a male class and the second correct class as a female class. Note that the first correct class may be the female class and the second correct class may be the male class.

[0127] The inference function 361 infers an unknown image using the learned model. The inference function 361 is an example of an inference means. Specifically, the inference function 361 infers a class to which a newly input image belongs using a machine learning model trained using a data set composed of images selected based on the relationship between a feature distribution regarding the feature amount of an image belonging to a first correct class and a feature distribution regarding the feature amount of an image belonging to a second correct class. For example, the inference function 361 infers whether the input image is an image of a man or an image of a woman using the learned model.

[0128] The trained model is, for example, a model that has been trained in advance as a machine learning model that classifies images containing both adults and children into males and females using a data set composed of images selected based on the relationship between a feature distribution related to the feature amounts of images belonging to a first correct answer class and a feature distribution related to the feature amounts of images belonging to a second correct answer class. Here, the contents that the image processing device 30 infers using the selected images and the trained model will be described with reference to FIG. 30. FIG. 30 is a schematic diagram for explaining the contents that the image processing device 30 infers according to the sixth embodiment.

[0129] For example, the inference function 361 of the image processing device 30 infers whether the input image 182 is an image of a man or an image of a woman using the learned model 181 (inference process 183 shown in FIG. 30). Then, the inference function 361 infers that the input image 182 is an image of a man as an inference result 184.

[0130] Next, a process executed by the image processing device 30 according to the sixth embodiment will be described below. Fig. 31 is a flowchart showing an example of a process executed by the image processing device 30 according to the sixth embodiment.

[0131] The inference function 361 reads a trained model and an image, and infers the class to which the read image belongs (step S61). When the processing circuit 350 finishes step S61, this process ends.

[0132] As described above, the image processing device 30 according to the sixth embodiment infers the class to which a newly input image belongs by using a machine learning model trained using a dataset consisting of images selected based on the relationship between the feature distribution of the feature amounts of images belonging to the first correct class and the feature distribution of the feature amounts of images belonging to the second correct class. As a result, for example, when images of a certain class in a dataset are biased toward images that are more dissimilar than images that are similar in appearance to images of another class, the image processing device 30 can improve the discrimination performance of the trained model for images that are similar in appearance to images of another class and reduce misclassification.

[0133] (Modification) In the above-mentioned embodiment, the target image classification is described as a form of classifying a tissue specimen image of a dissected lymph node into normal and lesion, and a form of classifying an image of a mixture of adults and children into male and female, but is not limited thereto. For example, the target image classification may be a form of classifying a brain PET (Positron Emission Tomography) image into normal and lesion.

[0134] Tumors near the brain include skull metastases and brain tumors. PET tends to accumulate in tumor cells, but in the case of the brain, it also tends to accumulate in normal tissue. On the other hand, in the case of skull metastases, PET accumulates in a fluffy pattern on the outside of the brain, and such accumulation is unlikely to occur in images of normal brains. Therefore, when tumor images are biased toward skull metastases, the tumor class will have fewer images similar to the normal class and more dissimilar images, so the effectiveness of the image selection process using this method can be expected.

[0135] According to at least one of the embodiments described above, when images of a certain class in a dataset are biased toward images that are more dissimilar than images that are similar in appearance to images of another class, it is possible to improve the discrimination performance of the trained model for images that are similar in appearance to images of another class and reduce misclassification.

[0136] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0137] 10 1st storage device 20 2nd storage device 30 Image processing device 40 Network 50 Third storage device 60 4th storage device 100 Information Processing Systems 310 NW (Network) Interface 320 Input Interface 330 Display 340 Memory circuit 350 Processing Circuit 351 Image Acquisition Function 352 Correct class acquisition function 353 Feature Acquisition Function 354 Selection Function 355 Probability Determination Function 356 Data Extensions 357 Grouping Function 358 Image number determination function 359 Dataset Configuration Functions 360 Learning Features 361 Inference Function 362 Bias determination function 363 Instruction acquisition function

Claims

1. image capture means for capturing a plurality of images; a correct class acquisition means for acquiring a correct class to which each of the plurality of images belongs; A feature acquisition means for acquiring feature amounts of each of the plurality of images; a selection means for selecting an input image to be used for machine learning from the images based on a relationship between a feature distribution for the feature amount of each image belonging to a first correct class and a feature distribution for the feature amount of each image belonging to a second correct class; An information processing device comprising:

2. the selection means selects, from among the images belonging to the first correct class, images distributed near the second correct class in preference to images distributed far from the second correct class; The information processing device according to claim 1 .

3. a probability determining means for determining a probability that each image will be selected based on a relationship between a feature distribution for the feature amount of each image belonging to the first correct class and a feature distribution for the feature amount of each image belonging to the second correct class; The selection means selects an input image based on the probability. The information processing device according to claim 2 .

4. a grouping means for classifying the images belonging to the first correct class into a plurality of groups based on the feature amount, the selection means selects an input image from each of the groups based on a relationship between a feature distribution for the feature amount of each of the images belonging to the second correct class and a feature distribution for the feature amount of the images belonging to each of the groups. The information processing device according to claim 1 .

5. the selection means selects, from among the plurality of groups, images belonging to a group distributed near the second correct class in preference to images belonging to a group distributed far from the second correct class; The information processing device according to claim 4.

6. a probability determining means for determining a probability of selecting an input image from each of the groups based on a relationship between a feature distribution for the feature amount of each of the images belonging to the second correct class and a feature distribution for the feature amount of the images belonging to each of the groups; said selecting means selecting an input image from each of said groups based on said probability; The information processing device according to claim 4.

7. and a number-of-images determining means for determining a number of input images to be selected from each of the groups based on a relationship between a feature distribution for the feature amount of each of the images belonging to the second correct class and a feature distribution for the feature amount of the images belonging to each of the groups, the selection means selects the number of input images from each of the groups; The information processing device according to claim 4.

8. The image number determination means further determines the number of images based on the number of the images belonging to each of the groups. The information processing device according to claim 7.

9. the number of images determining means determines the number of images to be selected from a group distributed near the second correct class among the plurality of groups so that the number of images is equal to or greater than the number of images to be selected from a group distributed far from the second correct class; 9. The information processing device according to claim 7 or 8.

10. Further comprising a learning means for learning a machine learning model using a data set constructed based on the input image selected by the selection means. The information processing device according to claim 1 .

11. The input image selected by the selection means is further provided with a data extension means for data extension of the input image, the selection means selects input images such that the sum of the input images selected from each group and the images whose data has been extended by the data extension means is the number of images; The learning means learns a machine learning model using a data set constructed based on the input image and an image data-extended by the data extension means. The information processing device according to claim 10.

12. The method further includes an inference means for inferring a class to which a newly input image belongs, using the learned machine learning model. The information processing device according to claim 10.

13. The method further includes a bias determination means for determining whether the feature distribution of the images belonging to the first correct class is biased toward either the images distributed near the group of the second correct class or the images distributed far away from the group of the second correct class; When the bias determination means determines that the images are biased, the selection means performs an image selection process; If the bias determination means determines that the images are not biased, all images belonging to the first correct class are selected as input images to be used in machine learning. The information processing device according to claim 1 .

14. The image processing device further includes an instruction acquisition unit for acquiring an instruction as to whether or not to execute an image selection process by the selection unit, When the bias determination means determines that the image is biased and the instruction acquisition means acquires an instruction to select an image, the selection means executes an image selection process; When the bias determination means determines that the image is not biased, or when the instruction acquisition means does not acquire an instruction to select an image, all images belonging to the first correct class are selected as input images to be used in machine learning. The information processing device according to claim 13.

15. an inference means for inferring a class to which a newly input image belongs, using a machine learning model trained using a data set composed of images selected based on a relationship between a feature distribution regarding the feature amounts of images belonging to a first correct class and a feature distribution regarding the feature amounts of images belonging to a second correct class; An information processing device comprising:

16. image capture means for capturing a plurality of images; a correct class acquisition means for acquiring a correct class to which each of the plurality of images belongs; A feature acquisition means for acquiring feature amounts of each of the plurality of images; a selection means for selecting an input image to be used for machine learning from the images based on a relationship between a feature distribution for the feature amount of each image belonging to a first correct class and a feature distribution for the feature amount of each image belonging to a second correct class; A medical device comprising:

17. acquiring a plurality of images; obtaining a correct class to which each of the plurality of images belongs; acquiring feature amounts of each of the plurality of images; selecting an input image to be used for machine learning from the images based on a relationship between a feature distribution for the feature amount of each image belonging to a first correct class and a feature distribution for the feature amount of each image belonging to a second correct class; An information processing method.

18. On the computer, Acquiring a plurality of images; A process of acquiring a correct class, which is a class to which each image belongs, from among the plurality of images; acquiring feature amounts of each of the plurality of images; selecting an input image to be used for machine learning from the images based on a relationship between a feature distribution for the feature amount of each image belonging to a first correct class and a feature distribution for the feature amount of each image belonging to a second correct class; A program for executing the above.

Images