Training apparatus, training method, and training program

Abstract

To realize a training apparatus capable of training a machine learning model so that inference accuracy is higher.SOLUTION: A training apparatus includes: a selection section that selects, on the basis of a label assigned to an image of interest included in a plurality of images and a label assigned to each of the plurality of images, each of the plurality of images as a positive sample, a hard negative sample, or an easy negative sample; a reselection section that reselects, on the basis of at least one of a similarity between features of the image of interest and features of the hard negative sample and a similarity between the features of the image of interest and features of the easy negative sample, each of the plurality of images selected by the selection section as the positive sample, the hard negative sample, or the easy negative sample; and a training section that uses the plurality of images reselected by the reselection section to train a machine learning model. Results of classification performed via the machine learning model assist in decision making in diagnosis made by a doctor or the like.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, and a learning program. [Background technology]

[0002] A technique for training a machine learning model using contrastive learning is known. In contrastive learning, a machine learning model is trained so that the inner product of the feature vector of an anchor, which is an image of interest, and the feature vector of a positive example becomes large, and the inner product of the feature vector of the anchor and the feature vector of a negative example becomes small. Patent Document 1 discloses a method for generating a machine learning model using SimCLR (A Sample Framework for Contrastive Learning of Visual Representations), which is an example of contrastive learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-178892 Summary of the Invention [Problem to be solved by the invention]

[0004] In contrastive learning, the degree of similarity between the anchor features and the negative example features may differ for each negative example, and some negative examples may have some commonalities with the anchor features, while others may have almost no commonalities with the anchor features. If contrastive learning is performed by uniformly defining multiple samples with different degrees of similarity to the anchor as negative examples, machine learning may not proceed properly, and the inference accuracy of the machine learning model may not be improved.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technique for training a machine learning model to achieve higher inference accuracy. [Means for solving the problem]

[0006] A learning device according to an exemplary aspect of the present disclosure includes a selection means that selects a plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and a label attached to each of the plurality of images; and a re-selection means that re-selects the plurality of images selected by the selection means into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between a feature amount of the image of interest and a feature amount of the weak negative examples and a similarity between a feature amount of the image of interest and a feature amount of the strong negative examples. and a learning means for training a machine learning model using the plurality of images reselected by the reselection means so that (i) the similarity between the feature values ​​of the image of interest and the feature values ​​of the positive examples increases, (ii) the similarity between the feature values ​​of the image of interest and the feature values ​​of the weak negative examples and the similarity between the feature values ​​of the image of interest and the feature values ​​of the strong negative examples decreases, and (iii) the similarity between the feature values ​​of the image of interest and the feature values ​​of the weak negative examples becomes greater than the similarity between the feature values ​​of the image of interest and the feature values ​​of the strong negative examples.

[0007] A learning method according to an exemplary aspect of the present disclosure includes a selection process in which at least one processor selects a plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and a label attached to each of the plurality of images; and a reclassification process in which the at least one processor reclassifies the plurality of images selected by the selection process into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between a feature amount of the image of interest and a feature amount of the weak negative examples and a similarity between a feature amount of the image of interest and a feature amount of the strong negative examples. and a learning process in which the at least one processor uses the plurality of images reselected by the reselection process to train a machine learning model so that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the weak negative examples and the similarity between the features of the image of interest and the features of the strong negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the weak negative examples becomes greater than the similarity between the features of the image of interest and the features of the strong negative examples.

[0008] A learning program according to an exemplary aspect of the present disclosure is a program for causing a computer to function as a learning device, the program including: a selection unit that selects a plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and a label attached to each of the plurality of images; and a classification unit that classifies the plurality of images selected by the selection unit into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between a feature amount of the image of interest and a feature amount of the weak negative examples and a similarity between a feature amount of the image of interest and a feature amount of the strong negative examples. The apparatus functions as a re-selection means for re-selecting the images into weak negative examples and strong negative examples, and a learning means for training a machine learning model using the plurality of images re-selected by the re-selection means so that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the weak negative examples and the similarity between the features of the image of interest and the features of the strong negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the weak negative examples becomes greater than the similarity between the features of the image of interest and the features of the strong negative examples. [Effects of the Invention]

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is provided in which a technology for training a machine learning model to achieve higher inference accuracy can be provided. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of a learning device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of a learning method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 4] FIG. 1 illustrates an example of classes and subclasses according to the present disclosure. [Figure 5] FIG. 1 illustrates an example of a class, intermediate class, and subclass according to the present disclosure. [Figure 6]FIG. 1 is a block diagram illustrating a configuration of a machine learning model according to the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating a specific example of a re-sorting process according to the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as a learning device and an information processing device according to the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating a specific example of a learning process according to the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] First Exemplary Embodiment A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Learning device configuration) The configuration of the learning device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the learning device 1. As shown in Fig. 1, the learning device 1 includes a selection unit 11, a re-selection unit 12, and a learning unit 13.

[0014] The selection unit 11 selects the images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the images and labels attached to each of the images. The re-selection unit 12 re-selects the images selected by the selection unit 11 into the positive examples, weak negative examples, and strong negative examples based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and the similarity between the feature amount of the image of interest and the feature amount of the strong negative examples. The learning unit 13 uses the multiple images reselected by the reselection unit 12 to train a machine learning model so that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the weak negative examples and the similarity between the features of the image of interest and the features of the strong negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the weak negative examples becomes greater than the similarity between the features of the image of interest and the features of the strong negative examples.

[0015] (Effect of learning device) As described above, the learning device 1 includes a selection unit 11 that selects a plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and a label attached to each of the plurality of images; a re-selection unit 12 that re-selects the plurality of images selected by the selection unit 11 into the positive examples, weak negative examples, and strong negative examples based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and the similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; and a learning unit 13 that trains a machine learning model using the plurality of images reselected by the separate unit 12 so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples. Therefore, the learning device 1 has the effect of training a machine learning model to achieve higher inference accuracy.

[0016] (Learning method flow) The flow of learning method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of learning method S1. As shown in Fig. 2, learning method S1 includes a selection process S11, a re-selection process S12, and a learning process S13.

[0017] In the selection process S11, at least one processor selects the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images. In the re-selection process S12, the at least one processor re-selects the plurality of images selected in the selection process S11 into the positive examples, weak negative examples, and strong negative examples based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and the similarity between the feature amount of the image of interest and the feature amount of the strong negative examples. In the learning process S13, the at least one processor uses the multiple images reselected by the reselection process S12 to train a machine learning model so that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the weak negative examples and the similarity between the features of the image of interest and the features of the strong negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the weak negative examples becomes greater than the similarity between the features of the image of interest and the features of the strong negative examples.

[0018] (Effect of learning method) As described above, the learning method S1 includes a selection process S11 in which at least one processor selects a plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images; and a process in which the at least one processor reselects the plurality of images selected by the selection process S11 into the positive examples, weak negative examples, and strong negative examples based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and the similarity between the feature amount of the image of interest and the feature amount of the strong negative examples. The configuration includes a re-screening process S12, and a learning process in which the at least one processor uses the multiple images re-screened by the re-screening process S12 to train a machine learning model so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples. Therefore, the learning method S1 has the effect of training a machine learning model to achieve higher inference accuracy.

[0019] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0020] <1. Configuration of Information Processing Device 1A> The information processing device 1A according to the present disclosure is a learning device that trains a machine learning model for performing an image recognition task. The information processing device 1A also performs an image recognition task using a machine learning model generated by machine learning. An example of an image recognition task is a classification task that identifies to which predefined class an object included as a subject in an image belongs. One example of the object included as a subject in an image is a specimen cell. In this classification task, the image to be recognized is classified into a class depending on whether the specimen cell is benign or malignant, and further into a subclass depending on the type of specimen cell. For example, the information processing device 1A can be used for cytology in rapid on-site evaluation (ROSE). The classification results obtained by the machine learning model assist, for example, physicians in making diagnostic decisions.

[0021] The configuration of the information processing device 1A will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A.

[0022] (Communications Department) The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0023] (Input section) The input unit 40A is configured to receive input to the information processing device 1A, and includes, for example, input devices such as a keyboard, a mouse, a touch panel, a camera, a microphone, etc. The input unit 40A may also be configured to receive data from the input devices via an interface such as a USB (Universal Serial Bus).

[0024] (output section) The output unit 50A is a component for performing output from the information processing device 1A, and includes, for example, output devices such as a display, a printer, a touch panel, a speaker, etc. The output unit 50A may also be configured to include, for example, an interface such as a USB, and to output data to the output device via the interface.

[0025] (Storage part) The storage unit 20A stores various types of information referenced by the control unit 10A. Examples of such information include an image set IS and a machine learning model LM1 used for machine learning training. Note that storing the machine learning model LM1 in the storage unit 20A means that parameters defining the machine learning model LM1 are stored in the storage unit 20A.

[0026] (Image set) The image set IS is a set of images used for training the machine learning model LM1. As an example, the images included in the image set IS are images that include objects such as specimen cells as subjects. Each image included in the image set IS is labeled with a label indicating the class and subclass to which the image belongs. In other words, each image included in the image set IS belongs to one of multiple classes, and also belongs to one of multiple subclasses obtained by further classifying each of the multiple classes. The classes and subclasses are classified according to the characteristics of the subjects included in the images.

[0027] FIG. 4 is a diagram showing an example of classes and subclasses. FIG. 4 shows that specimen cells included as subjects in an image are classified into classes depending on whether they are benign or malignant. FIG. 4 also shows that specimen cells included as subjects in an image are classified into subclasses depending on the type of specimen cell. In FIG. 4, for example, the class "benign cells" is classified into multiple subclasses: "EC normal epithelial cells," "IEC inflammatory ECs," "M macrophages," "LC lymphocytes," and "WBC white blood cells."

[0028] Furthermore, an image may belong to one of a plurality of middle classes, each of which is a group of one or more subclasses, where each class is divided into a plurality of middle classes. An example in which the example shown in Figure 4 is further divided into middle classes is shown in Figure 5. Figure 5 is a diagram showing an example of classes, middle classes, and subclasses.

[0029] In Figure 5, for example, the class "benign cells" is classified into three middle classes: "normal cells," "normal cells with findings," and "other normal cells." Also in Figure 5, the middle class "normal cells" is grouped with the subclass "EC normal epithelial cells," and the middle class "normal cells with findings" is grouped with the subclasses "IEC inflammatory EC" and "M macrophage."

[0030] (machine learning model) The machine learning model LM1 is a model generated by machine learning, such as a neural network. As an example, the machine learning model LM1 includes a first layer group that receives an image as an input and generates features of the image, and a second layer group that is connected to the first layer group and receives the image features as an input and classifies the input image into the class or subclass.

[0031] FIG. 6 is a diagram showing an example of the machine learning model LM1. In the example of FIG. 6, the machine learning model LM1 includes a feature analysis model LM11 and a classifier LM12. The feature analysis model LM11 is an encoder (first layer group) that receives an input image and generates features of the input image. The feature analysis model LM11 is used as a pre-learning model for the classifier LM12. When an input image including a subject is input, the feature analysis model LM11 outputs features (feature vectors) of the input image.

[0032] The classifier LM12 is a classifier (second layer group) that is connected to the feature analysis model LM11 and classifies the input image into a class or subclass using the feature quantities of the input image output from the feature analysis model LM11 as input. The machine learning model LM1 outputs the class or subclass classified by the classifier as the classification result.

[0033] (Control unit) 3, the control unit 10A includes a learning phase execution unit 110A and an estimation phase execution unit 120A. The learning phase execution unit 110A includes an acquisition unit 111A, a sample selection unit 112A, a reselection unit 113A, and a learning unit 114A. The estimation phase execution unit 120A includes a classification unit 121A. The sample selection unit 112A, the reselection unit 113A, the learning unit 114A, and the classification unit 121A are examples of the selection means, the reselection means, the learning means, and the classification means according to the present disclosure, respectively.

[0034] (Acquisition Department) The acquisition unit 111A acquires the image set IS. As an example, the acquisition unit 11A acquires the image set IS by receiving the image set IS from another device via the communication unit 30A. The acquisition unit 11A may also acquire the image set IS input to the input unit 40A. The acquisition unit 111A may also acquire the image set IS by reading the image set IS from a storage destination specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A).

[0035] (Sample Selection Department) The sample selection unit 112A selects an arbitrary anchor (image of interest) from the image set IS, and identifies the sample type of each of the other images included in the image set IS based on the label attached to the selected anchor and the labels attached to the other images included in the image set IS. The sample type indicates the selection result for contrastive learning of the sample image. Examples of sample types include a positive sample, a hard negative sample, and an easy negative sample.

[0036] A positive example is an image that belongs to the same subclass as the subclass to which the anchor belongs. A positive example may also be a data-augmented image obtained by data-augmenting an anchor. Examples of data-augmented images include an image in which the anchor is rotated, an image in which the subject included in the anchor is moved, an image in which the subject included in the anchor is enlarged or reduced, an image in which the anchor is flipped vertically or horizontally, an image in which a portion of the anchor is cropped, and an image in which the hue, saturation, or brightness of the anchor is changed.

[0037] A weak negative example is an image that belongs to a subclass different from the subclass to which the anchor belongs, but belongs to the same class as the anchor. A strong negative example is an image that belongs to a class different from the class to which the anchor belongs. It can also be said that a weak negative example is a sample that is more difficult to distinguish from a positive example than a strong negative example.

[0038] As an example, the sample selection unit 112A selects, as a positive example, an image that belongs to the same subclass as the subclass to which the anchor belongs. Furthermore, the sample selection unit 112A selects, as a weak negative example, an image that belongs to a subclass different from the subclass to which the anchor belongs and also belongs to the same class to which the anchor belongs. Furthermore, the sample selection unit 112A selects, as a strong negative example, an image that belongs to a class different from the class to which the anchor belongs. Furthermore, the sample selection unit 112A may select, as a positive example, a data-augmented image obtained by data-augmenting an anchor.

[0039] Furthermore, the sample selection unit 112A may further classify the weak negative examples into a plurality of sample types. For example, when images are classified into a middle class in addition to classes and subclasses, the sample selection unit 112A may classify, among the images selected as the weak negative examples, an image that belongs to the same middle class as the middle class to which the anchor belongs as a first weak negative example, and an image that belongs to a different middle class from the middle class to which the anchor belongs as a second weak negative example.

[0040] (Re-sorting Department) The re-selection unit 113A re-selects the multiple images that have been selected into multiple sample types by the sample selection unit 112A based on the feature amounts of each image, etc. More specifically, the re-selection unit 113A re-selects the multiple images selected by the sample selection unit 112A into positive examples, weak negative examples, and strong negative examples based on at least one of the similarity between the feature amounts of the anchors and the feature amounts of the weak negative examples and the similarity between the feature amounts of the anchors and the feature amounts of the strong negative examples.

[0041] Here, the similarity between the feature of the anchor and the feature of the sample (positive example / weak negative example / strong negative example) can be, for example, a distance (Euclidean distance, etc.) in a predetermined feature space. In this case, the re-selection unit 113A can be said to re-select multiple images based on at least one of the distance between the anchor and the weak negative example and the distance between the anchor and the strong negative example in the predetermined feature space. More specifically, as an example, the re-selection unit 113A re-selects multiple images based on a threshold determined by at least one of (i) the confidence interval and maximum value of the distance between the anchor and the weak negative example, and (ii) the confidence interval and minimum value of the distance between the anchor and the strong negative example.

[0042] However, the similarity between the feature amount of the anchor and the feature amount of the sample is not limited to the above example, and may be, for example, the inner product of a feature vector representing the feature amount of the anchor and a feature vector representing the feature amount of the sample, or cosine similarity.

[0043] Furthermore, when weak negative examples are further sorted into a plurality of sample types, the re-sorting unit 113A may perform the re-sorting based on the similarity between each sample type and the anchor. For example, when weak negative examples are sorted into a first weak negative example and a second weak negative example, the re-sorting unit 113A may re-sort the plurality of images sorted by the sample sorting unit 112A into positive examples, a first weak negative example, a second weak negative example, and a strong negative example based on at least one of the similarity between the feature of the anchor and the feature of the first weak negative example, the similarity between the feature of the anchor and the feature of the second weak negative example, and the similarity between the feature of the anchor and the feature of the strong negative example.

[0044] (Study Department) The learning unit 114A trains the feature analysis model LM11 using the multiple images reselected by the reselection unit 113A. That is, the learning unit 114A updates the parameters that define the feature analysis model LM11, which are stored in the storage unit 20A. At this time, as an example, the learning unit 114A trains the feature analysis model LM11 so that (i) the similarity between the features of the anchors and the features of the positive examples increases, (ii) the similarity between the features of the anchors and the features of the weak negative examples decreases, (iii) the similarity between the features of the anchors and the features of the strong negative examples decreases, and (iv) the similarity between the features of the anchors and the features of the weak negative examples increases beyond the similarity between the features of the anchors and the features of the strong negative examples.

[0045] Furthermore, the learning unit 114A uses an image set SI in which each image is labeled to train a machine learning model LM1 in which a classifier LM12 is connected to a feature analysis model LM11. That is, the learning unit 114A updates the parameters that define the machine learning model LM1, which are stored in the storage unit 20A. The training process performed by the learning unit 114A will be described in detail later.

[0046] (Classification section) The classifying unit 121A acquires a target image, which is an image including a subject that is the recognition target of the image recognition task, and inputs the acquired target image to the machine learning model LM1, thereby classifying the input target image into a class or subclass. As an example, the classifying unit 121A acquires the target image by receiving the target image from another device via the communication unit 30A. Alternatively, the classifying unit 121A may acquire the target image input to the input unit 40A. Alternatively, the classifying unit 121A may acquire the target image by reading it from a storage destination specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A).

[0047] Furthermore, the classifying unit 121A outputs the classification result. As an example, the classifying unit 121A may output the classification result by writing it to a storage destination (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Furthermore, the classifying unit 121A may transmit the classification result via the communication unit 30A, or may output the classification result to an output device such as a display.

[0048] <2. Processing example by the re-sorting unit> A specific example of the re-selection process performed by the re-selection unit 113A will be described. Specific examples of the re-selection process performed by the re-selection unit 113A include (i) a process of changing the sample type (positive example, weak negative example, strong negative example), and (ii) a process of excluding a sample from the image set used for learning. These methods will be described in order.

[0049] (Re-screening process example 1: changing sample type) In this example, the re-selection unit 113A changes the sample type of an image whose feature amount is close to that of a strong negative example, among the images whose sample type is a weak negative example, to a strong negative example. In other words, in this example, the re-selection unit 113A selects, among the images selected as weak negative examples by the sample selection unit 112A, an image whose distance from the anchor is greater than a threshold, as a strong negative example.

[0050] More specifically, the re-selection unit 113A first calculates a 95% confidence interval of the distance Dis_AH from the anchor to the weak negative example. The distance Dis_AH is calculated, for example, by the following formula (1). In formula (1), H is a set of weak negative examples, and h∈H is a weak negative example included in the set H. Dis_AH h is the Euclidean distance between the anchor and the weak negative example h in a given feature space.

number

[0051] Next, the re-selection unit 113A sets the upper limit of the 95% confidence interval of the distance Dis_AH as a threshold th, and selects the images selected as weak negative examples with the distance Dis_AH h The re-selection unit 113A changes the sample type of an image whose distance from the anchor is smaller than the threshold value th from a weak negative example to a strong negative example, among the images selected as strong negative examples by the sample selection unit 112A.

[0052] 7 is a diagram showing a specific example of the re-screening process. In the example of FIG. 7, the re-screening unit 113A changes the sample types of samples s1 and s2, which are strong negative examples whose distances from the anchor are smaller than the threshold th, from strong negative examples to weak negative examples. In addition, the re-screening unit 113A changes the sample types of samples s1 and s2, which are strong negative examples whose distances from the anchor are smaller than the threshold th, from strong negative examples to weak negative examples. h The sample type of sample s3, which is a weak negative example where σ is greater than the threshold th, is changed from a weak negative example to a strong negative example.

[0053] (Re-screening processing example 2: Exclusion) In this example, the re-selection unit 113A excludes from learning the feature analysis model LM11, among the images selected as weak negative examples by the sample selection unit 112A, images whose distance from the anchor is greater than the threshold th. In other words, the re-selection unit 113A selects from among the images selected as weak negative examples by the sample selection unit 112A, images whose distance from the anchor is greater than the threshold th as examples not to be used in learning the feature analysis model LM11.

[0054] Furthermore, the re-selection unit 113A excludes from learning the feature analysis model LM11, among the images selected as strong negative examples by the sample selection unit 112A, images whose distance from the anchor is smaller than the threshold th. In other words, the re-selection unit 113A selects, among the images selected as strong negative examples by the sample selection unit 112A, images whose distance from the anchor is smaller than the threshold th as examples not to be used in learning the feature analysis model LM11.

[0055] (Another example of a threshold used for re-screening) The threshold th is not limited to the upper limit of the 95% confidence interval of the distance Dis_AH, and may be other values. For example, the threshold th is the distance Dis_AH between the anchor and the weak negative example h. h , and for example, the distance Dis_AE between the anchor and the strong negative example n∈N (N is a set of strong negative examples). n The threshold th may be the minimum value of the distance Dis_AE(Σ n∈N Dis_AE n The threshold value th may be the lower limit of the 95% confidence interval of the distance Dis_AH. The threshold value th may also be the average value of the upper limit of the 95% confidence interval of the distance Dis_AH and the upper limit of the 95% confidence interval of the distance Dis_AE.

[0056] (Example of re-screening process when multiple weak negative examples are selected) Furthermore, when weak negative examples are further sorted into a plurality of sample types, the re-sorting unit 113A may perform the re-sorting based on the similarity between each sample type and the anchor. For example, when weak negative examples are sorted into a first weak negative example and a second weak negative example, the re-sorting unit 113A may, for example, set the upper limit of the 95% confidence interval of the distance Dis_AH1 from the anchor to the first weak negative example as a threshold th1, and change the first weak negative example whose distance to the anchor is greater than the threshold th1 to the second weak negative example. Furthermore, the re-sorting unit 113A may, for example, change the second weak negative example whose distance to the anchor is less than the threshold th1 to the first weak negative example. The threshold th1 is not limited to the upper limit of the 95% confidence interval of the distance Dis_AH1, and may be, for example, (i) the upper limit of the 95% confidence interval of the distance Dis_AH1 from the anchor to the first weak negative example. h1(ii) the lower limit of the confidence interval of the distance Dis_AH2 from the anchor to the second weak negative example; (iii) the distance Dis_AH2 from the anchor to the second weak negative example. h2 (iv) the average value between the upper limit of the 95% confidence interval of the distance Dis_AH1 and the lower limit of the 95% confidence interval of the distance Dis_AH2, etc.

[0057] Further, as an example, the re-selection unit 113A may set the upper limit of the 95% confidence interval of the distance Dis_AH2 from the anchor to the second weak negative example as a threshold th2, and change the second weak negative example whose distance to the anchor is greater than the threshold th2 to a strong negative example, and change the strong negative example whose distance to the anchor is less than the threshold th2 to a second weak negative example. The threshold th2 is not limited to the upper limit of the 95% confidence interval of the distance Dis_AH2, and may be, for example, (i) the distance Dis_AH2 h2 (ii) the lower limit of the confidence interval of the distance Dis_AE from the anchor to the strong negative example, (iii) the distance Dis_AE n (iv) the average value between the upper limit of the 95% confidence interval of the distance Dis_AH2 and the lower limit of the confidence interval of the distance Dis_AE, etc.

[0058] <3. Processing example by the learning unit> Next, the process by the learning unit 114A of (a) training the feature analysis model LM11 and (b) training the machine learning model LM1 in which the classifier LM12 is connected to the feature analysis model LM11 will be described.

[0059] (Processing example 1 for training feature analysis model LM11) In this example, the learning unit 114A trains the machine learning model LM1 using a loss function whose contribution changes depending on whether the image is a positive example, a weak negative example, or a strong negative example. As an example, the learning unit 114A trains the feature analysis model LM11 so that (i) the similarity (such as the inner product of feature vectors) between the feature of the anchor and the feature of the positive example increases, (ii) the similarity between the feature of the anchor and the feature of the weak negative example decreases, (iii) the similarity between the feature of the anchor and the feature of the strong negative example decreases, and (iv) the similarity between the feature of the anchor and the feature of the weak negative example increases more than the similarity between the feature of the anchor and the feature of the strong negative example.

[0060] As an example, the learning unit 114A trains the feature analysis model LM11 so that the value of the following loss function, Equation (2), becomes small.

number

[0061] To reduce the value of equation (2), which is a loss function, the learning unit 114A trains the feature analysis model LM11 so that the inner product between the feature vector of the anchor and the feature vector of the positive example becomes larger. The learning unit 114A also trains the feature analysis model LM11 so that the inner product between the feature vector of the anchor and the feature vector of the weak negative example becomes smaller. The learning unit 114A also trains the feature analysis model LM11 so that the inner product between the feature vector of the anchor and the feature vector of the strong negative example becomes smaller.

[0062] That is, in equation (2), the feature is a feature vector, and the similarity between the anchor feature vector and each of the feature vectors of the positive, weak negative, and strong negative examples is based on the dot product of the anchor feature vector and each of the feature vectors of the positive, weak negative, and strong negative examples.

[0063] In equation (2), the weighting coefficient W given to weak negative examples is h and the weighting factor W given to the strong negative examples n The sum of these is 1.0. Also, the weighting coefficient W h is the weighting factor W n is set to a value larger than the value of the anchor feature vector. Therefore, the value of equation (2) changes more greatly depending on the inner product of the anchor feature vector and the weak negative example feature vector than on the inner product of the anchor feature vector and the strong negative example feature vector. In other words, the learning unit 114A trains the feature analysis model LM11 so that the similarity between the anchor feature vector and the weak negative example feature vector is greater than the similarity between the anchor feature vector and the strong negative example feature vector.

[0064] (Processing example 2 for training feature analysis model LM11) As described above, the images included in the image set IS may be classified into a plurality of middle classes, each of which is a group of one or more subclasses. In this case, in addition to the positive examples, weak negative examples, and strong negative examples described above, the weak negative examples are also classified into either the first weak negative example or the second weak negative example.

[0065] In this case, the learning unit 114A trains the feature analysis model LM11 so that the similarity between the feature of the anchor and the feature of the first weak negative example is greater than the similarity between the feature of the anchor and the feature of the second weak negative example. The learning unit 114A trains the feature analysis model LM11 so that the value of the following loss function, Equation (3), becomes small.

number

[0066] To reduce the value of equation (3), which is a loss function, the learning unit 114A trains the feature analysis model LM11 so that the inner product of the anchor's feature vector and the feature vector of the first weak negative example is reduced. Furthermore, the learning unit 114A trains the feature analysis model LM11 so that the inner product of the anchor's feature vector and the feature vector of the second weak negative example is reduced. Furthermore, the learning unit 114A trains the feature analysis model LM11 so that the inner product of the anchor's feature vector and the feature vector of the positive example is increased. Furthermore, for strong negative examples, the learning unit 114A trains the feature analysis model LM11 so that the inner product of the anchor's signal vector and the feature vector of the strong negative example is reduced.

[0067] In equation (3), the weighting factor W assigned to the first weak negative example is h1 and the weighting factor W given to the second weak negative example h2 and the weighting factor W given to the strong negative examples. n The sum of these is 1.0. Also, the weighting coefficient W h1 and weighting factor W h2 is the weighting factor W n Therefore, the value of formula (3) changes more depending on the dot product of the anchor's feature vector and the feature vectors of the weak negative examples (first and second weak negative examples) than on the dot product of the anchor's feature vector and the feature vector of the strong negative example. That is, the learning unit 114A learns to move the first weak negative example and the second weak negative example further away from the anchor.

[0068] In addition, in equation (3), the weighting coefficient W h1 is the weighting factor W h2 may be set to a larger value. In this case, the value of equation (3) changes more depending on the inner product of the feature vector of the anchor and the feature vector of the first weak negative example than on the inner product of the feature vector of the anchor and the feature vector of the second weak negative example. That is, the learning unit 114A trains the feature analysis model LM11 so as to move the first weak negative example further away from the anchor. In other words, the learning unit 114A trains the feature analysis model LM11 so that the similarity between the feature of the anchor and the feature of the first weak negative example is greater than the similarity between the feature of the anchor and the feature of the second weak negative example.

[0069] (Example of a process for training the machine learning model LM1) Furthermore, the learning unit 114A trains the machine learning model LM1 using a plurality of images and teacher labels corresponding to the results of classifying each of the plurality of images into a class or subclass. The method and loss function (second loss function) used by the learning unit 114A to train the machine learning model LM1 are known methods. One example is a method that uses cross-entropy to minimize the error between the output data from the machine learning model LM1 and the teacher labels. The learning unit 114A mainly trains the classifier LM12, but may also train the feature analysis model LM11 so as to fine-tune it as needed.

[0070] <4. Effects of Information Processing Device 1A> FIG. 9 is a diagram showing an example of the learning results of a machine learning model when re-sorting is not performed. In the example of FIG. 9, distribution D11 shows an example of the distribution of samples (images) in the initial stage of learning in a predetermined feature space, and distribution D12 shows the distribution of samples after learning in the feature space. As shown in distribution D11, the anchor and positive examples are located close to each other in the feature space. On the other hand, among the weak negative examples, not only are there some that are as close to the anchor as positive examples, but there are also some that are as far from the anchor as strong negative examples. In this case, the weak negative examples as a whole may be mistakenly recognized as being sufficiently far from the anchor before learning, which may result in inappropriate learning.

[0071] Additionally, in distribution D11, while strong negative examples are generally far from the anchors, some are as close to the anchors as weak negative examples. This may be because the strong negative examples happen to have characteristics similar to those of weak negative examples, or because they were mislabeled. There are also cases where doctors' judgments differ, and the labels assigned are not necessarily appropriate. When a machine learning model is trained using such images, the strong negative examples may not be far enough from the anchors, as shown in distribution D12.

[0072] In contrast, according to the information processing device 1A of the present disclosure, even if the training data includes images that would be classified as strong negative examples based on the label information but are actually relatively close to the anchor (weak negative examples in terms of distance from the anchor in feature space), or images that would be classified as weak negative examples based on the label information but are actually relatively far from the anchor (strong negative examples in terms of distance from the anchor in feature space), the re-selection unit 113A can re-select images, thereby allowing the feature analysis model LM11 to be trained more appropriately.Furthermore, according to the information processing device 1A, even if the training data includes inaccurate label information, the re-selection unit 113A can re-select images, thereby allowing the feature analysis model LM11 to be trained more appropriately.

[0073] Furthermore, according to the information processing device 1A of the present disclosure, the feature analysis model LM11 is trained so that the similarity between the feature of the anchor and the feature of the weak negative example is greater than the similarity between the feature of the anchor and the feature of the strong negative example, thereby further improving the classification accuracy of the machine learning model LM1.

[0074] Furthermore, in the information processing device 1A according to the present disclosure, the images included in the image set IS belong to one of a plurality of classes and also to one of a plurality of subclasses obtained by further classifying each of the plurality of classes, a label attached to an image indicates the class and subclass to which the image belongs, and the sample selection unit 112A selects, as positive examples, images that belong to the subclass to which the anchor belongs, selects, as weak negative examples, images that belong to a subclass different from the subclass to which the anchor belongs and that also belong to the class to which the anchor belongs, and selects, as strong negative examples, images that belong to a class different from the class to which the anchor belongs. Thus, according to the information processing device 1A, by using images that belong to classes and subclasses, the feature analysis model LM11 can be trained so as to improve the inference accuracy of the machine learning model LM1.

[0075] Furthermore, the information processing device 1A employs a configuration in which the re-selection unit 113A re-selects images included in the image set IS based on at least one of the distance between the anchor and a weak negative example in a predetermined feature space and the distance between the anchor and a strong negative example. Therefore, according to the information processing device 1A, the distance from the anchor in the feature space can be reflected in the selection of sample types, thereby enabling the feature analysis model LM11 to be trained more appropriately.

[0076] Furthermore, the information processing device 1A employs a configuration in which the re-selection unit 113A re-selects images included in the image set IS based on a threshold determined by at least one of (i) the confidence interval and maximum value of the distance between the anchor and the weak negative example, and (ii) the confidence interval and minimum value of the distance between the anchor and the strong negative example. Therefore, according to the information processing device 1A, the feature analysis model LM11 can be more appropriately trained by re-selecting images based on the confidence interval or upper limit value of the distance between the anchor and the weak negative example in the feature space.

[0077] Furthermore, the information processing device 1A employs a configuration in which the reselection unit 113A selects, from among the images selected as weak negative examples by the sample selection unit 112A, images whose distance from the anchor is greater than a predetermined threshold as strong negative examples. Therefore, according to the information processing device 1A, by changing the sample type of an image whose distance from the anchor is greater than the predetermined threshold from a weak negative example to a strong negative example, it is possible to prevent weak negative examples as a whole from being mistakenly recognized as being sufficiently far from the anchor in a pre-learning state.

[0078] Furthermore, the information processing device 1A employs a configuration in which the reselection unit 113A selects, from among the images selected as strong negative examples by the sample selection unit 112A, images whose distance from the anchor is smaller than a threshold value as weak negative examples. In this way, by changing the sample type of an image whose distance from the anchor is smaller than a predetermined threshold value from a strong negative example to a weak negative example, it is possible to prevent an event from occurring in which a strong negative example is not sufficiently separated from the anchor after learning.

[0079] Furthermore, the information processing device 1A employs a configuration in which the re-selection unit 113A selects, from among the images selected as weak negative examples by the sample selection unit 112A, images whose distance from the anchor is greater than a threshold value as examples not to be used in training the feature analysis model LM11. By excluding weak negative examples whose distance from the anchor is greater than a predetermined threshold value from training the feature analysis model LM11 in this way, it is possible to prevent weak negative examples as a whole from being mistakenly recognized as being sufficiently far from the anchor in a state before training.

[0080] Furthermore, the information processing device 1A employs a configuration in which the re-selection unit 113A selects, from among the images selected as strong negative examples by the sample selection unit 112A, images whose distance from the target image is smaller than a threshold value as examples not to be used in training the machine learning model. By excluding sample types of images whose distance from the anchor is smaller than a predetermined threshold value from training of the feature analysis model LM11 in this way, it is possible to prevent an event from occurring in which strong negative examples are not sufficiently separated from the anchor after training. The learning device according to any one of claims 3 to 5.

[0081] Furthermore, the information processing device 1A employs a configuration in which the feature quantities of samples used in training the feature analysis model LM11 are feature vectors, and the similarities between the feature vector of the anchor and each of the feature vectors of the positive, weak negative, and strong negative examples are based on the dot products of the feature vector of the anchor and each of the feature vectors of the positive, weak negative, and strong negative examples. Therefore, the information processing device 1A can train the feature analysis model LM11 based on the dot products of the feature vectors.

[0082] Furthermore, in the information processing device 1A, the images included in the image set IS are further divided into a plurality of middle classes obtained by classifying each of a plurality of classes into a plurality of classes, and belong to any of a plurality of middle classes obtained by grouping one or more of the subclasses. The sample selection unit 112A selects, among the weak negative examples, an image that belongs to the same middle class as the middle class to which the anchor belongs as a first weak negative example, and selects, among the weak negative examples, an image that belongs to a middle class different from the middle class to which the anchor belongs as a second weak negative example. The re-selection unit 113A compares the feature amount of the anchor and the feature amount of the first weak negative example. The information processing device 1A employs a configuration in which the following is true: the plurality of images selected by the sample selection unit 112A are re-sorted into positive examples, first weak negative examples, second weak negative examples, and strong negative examples based on at least one of the similarity between the features of the anchor and the features of the second weak negative example, the similarity between the features of the anchor and the features of the strong negative example, the similarity between the features of the anchor and the features of the first weak negative example, and the similarity between the features of the anchor and the features of the strong negative example. The learning unit 114A trains the feature analysis model LM11 so that the similarity between the features of the anchor and the features of the first weak negative example is greater than the similarity between the features of the anchor and the features of the second weak negative example. Therefore, according to the information processing device 1A, the feature analysis model LM11 is trained taking into account the first weak negative examples and the second weak negative examples that are further selected from the weak negative examples, thereby achieving the effect of training the feature analysis model LM11 to improve the inference accuracy of the machine learning model LM1.

[0083] Furthermore, the information processing device 1A employs a configuration in which subjects included in images used for learning are specimen cells, and the images are classified into classes depending on whether the specimen cells included in the images are benign or malignant, and into subclasses depending on the type of specimen cells included in the images. Therefore, the information processing device 1A has an effect of improving the inference accuracy of the machine learning model LM1 that classifies input images into classes indicating whether the specimen cells are benign or malignant, and into subclasses depending on the type of specimen cells.

[0084] Furthermore, the information processing device 1A employs a configuration in which the sample selection unit 112A selects, as a positive example, a data-augmented image obtained by data-augmenting an anchor. Therefore, according to the information processing device 1A, by performing machine learning using, as a positive example, a data-augmented image obtained by data-augmenting an anchor, the feature analysis model LM11 can be trained more appropriately.

[0085] Furthermore, in the information processing device 1A, the machine learning model LM1 includes a first layer group that receives an image as input and generates features of the image, and a second layer group that is connected to the first layer group and receives the image features as input and classifies the input image into classes or subclasses, and the learning unit 114A trains the machine learning model LM1 using multiple images and teacher labels corresponding to the results of classifying each of the multiple images into a class or subclass. Therefore, the information processing device 1A has the effect of being able to generate a machine learning model LM1 with higher inference accuracy.

[0086] Furthermore, the information processing device 1A is configured to include a class classification unit 121A that classifies an input image containing a subject into a class or subclass by inputting the image containing a subject into the machine learning model LM1. Therefore, the information processing device 1A has the effect of being able to more accurately classify an image containing a subject into a class or subclass by using the machine learning model LM1 generated by machine learning.

[0087] [Software implementation example] Some or all of the functions of the learning device 1 and information processing device 1A (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0088] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. Figure 8 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0089] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0090] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0091] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0092] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0093] Furthermore, the functions of each of the devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0094] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0095] (Appendix A1) a selection means for selecting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images; a re-sorting means for re-sorting the plurality of images sorted by the sorting means into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning means for training a machine learning model using the plurality of images reselected by the reselection means so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A learning device comprising:

[0096] (Appendix A2) each of the plurality of images belongs to one of a plurality of classes and also belongs to one of a plurality of subclasses obtained by further classifying each of the plurality of classes; the labels indicate a class and a subclass to which each of the plurality of images belongs; The selecting means is selecting an image belonging to the subclass to which the target image belongs as a positive example; selecting, as weak negative examples, images that belong to a subclass different from the subclass to which the target image belongs and that belong to the class to which the target image belongs; selecting an image belonging to a class different from the class to which the target image belongs as a strong negative example; 10. A learning device according to claim A1.

[0097] (Appendix A3) the re-selection means re-selects the plurality of images based on at least one of a distance between the target image and the weak negative examples and a distance between the target image and the strong negative examples in a predetermined feature space; 10. The learning device according to claim A1 or A2.

[0098] (Appendix A4) The re-selection means re-selects the plurality of images based on a threshold determined by at least one of (i) a confidence interval and a maximum value of a distance between the target image and the weak negative example, and (ii) a confidence interval and a minimum value of a distance between the target image and the strong negative example. 1. A learning device as described in Appendix A3.

[0099] (Appendix A5) The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold value as the strong negative examples. 1. A learning device as described in Appendix A3.

[0100] (Appendix A6) The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as the weak negative examples. 1. A learning device as described in Appendix A3.

[0101] (Appendix A7) The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold value as examples not to be used in training the machine learning model.

[0102] (Appendix A8) The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as examples not to be used in training the machine learning model.

[0103] (Appendix A9) the feature amount referred to by the learning means is a feature vector, the similarity between the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example is based on an inner product of the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example; 10. The learning device according to any one of appendices A1 to A8.

[0104] (Appendix A10) Each of the plurality of images further belongs to one of a plurality of middle classes obtained by classifying each of the plurality of classes into a plurality of middle classes, each middle class being a group of one or more of the subclasses; The selecting means is Among the weak negative examples, an image that belongs to the same middle class as the target image belongs to is defined as a first weak negative example; Among the weak negative examples, an image belonging to a middle class different from the middle class to which the target image belongs is defined as a second weak negative example; The re-sorting means re-sorting the plurality of images selected by the sorting means into the positive example, the first weak negative example, the second weak negative example, and the strong negative example based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the first weak negative example, the similarity between the feature amount of the image of interest and the feature amount of the second weak negative example, and the similarity between the feature amount of the image of interest and the feature amount of the strong negative example; The learning means training the machine learning model so that a similarity between the feature of the image of interest and the feature of the first weak negative example is greater than a similarity between the feature of the image of interest and the feature of the second weak negative example; 10. A learning device as described in Appendix A2.

[0105] (Appendix A11) the subject included in the image is a specimen cell, The image is classified into classes depending on whether the specimen cells contained in the image are benign or malignant, and into subclasses depending on the type of specimen cells contained in the image. 10. The learning device according to claim A2 or A10.

[0106] (Appendix A12) The selection means selects a data-augmented image obtained by data-augmenting the target image as a positive example. 12. The learning device according to any one of appendices A1 to A11.

[0107] (Appendix A13) The machine learning model is a first group of layers that receives an image as input and generates features of the image; a second layer group connected to the first layer group, which receives the image feature amount as an input and classifies the input image into the class or the subclass; Equipped with the learning means trains the machine learning model using a plurality of images and training labels corresponding to the results of classifying each of the plurality of images into the class or the subclass; 10. A learning device as described in Appendix A2.

[0108] (Appendix A14) a classification means for inputting an image including a subject into the machine learning model trained by the learning device according to any one of Supplementary Notes A1 to A13, and classifying the image into a class or a subclass; The classification device further comprises:

[0109] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0110] (Appendix B1) a sorting process in which at least one processor sorts the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label assigned to an image of interest included in the plurality of images and labels assigned to each of the plurality of images; a re-sorting process in which the at least one processor re-sorts the images sorted by the sorting process into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning process in which the at least one processor uses the plurality of images reselected by the reselection process to train a machine learning model such that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; Learning methods including.

[0111] (Appendix B2) each of the plurality of images belongs to one of a plurality of classes and also belongs to one of a plurality of subclasses obtained by further classifying each of the plurality of classes; the labels indicate a class and a subclass to which each of the plurality of images belongs; In the selection process, the at least one processor: selecting an image belonging to the subclass to which the target image belongs as a positive example; selecting, as weak negative examples, images that belong to a subclass different from the subclass to which the target image belongs and that belong to the class to which the target image belongs; selecting an image belonging to a class different from the class to which the target image belongs as a strong negative example; Study methods described in Appendix B1.

[0112] (Appendix B3) In the re-selection process, the at least one processor re-selects the plurality of images based on at least one of a distance between the target image and the weak negative examples and a distance between the target image and the strong negative examples in a predetermined feature space. Study methods described in Appendix B1 or B2.

[0113] (Appendix B4) In the re-selection process, the at least one processor re-selects the plurality of images based on a threshold determined by at least one of (i) a confidence interval and a maximum value of a distance between the image of interest and the weak negative example, and (ii) a confidence interval and a minimum value of a distance between the image of interest and the strong negative example. Study methods described in Appendix B3.

[0114] (Appendix B5) In the re-selection process, the at least one processor selects, from among the images selected as the weak negative examples by the selection process, images whose distance from the target image is greater than a threshold value as the strong negative examples. Study methods described in Appendix B3.

[0115] (Appendix B6) In the re-selection process, the at least one processor selects, from among the images selected as the strong negative examples by the selection process, images whose distance from the target image is smaller than a threshold value as the weak negative examples. Study methods described in Appendix B3.

[0116] (Appendix B7) In the re-selection process, the at least one processor selects, from among the images selected as the weak negative examples by the selection process, images whose distance from the target image is greater than a threshold, as examples not to be used in training the machine learning model. Study methods described in Appendix B3.

[0117] (Appendix B8) In the re-selection process, the at least one processor selects, from among the images selected as the strong negative examples by the selection process, images whose distance from the target image is smaller than a threshold, as examples not to be used in training the machine learning model. Study methods described in Appendix B3.

[0118] (Appendix B9) the feature amounts referred to in the learning process are feature vectors, the similarity between the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example is based on an inner product of the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example; A study method described in one of Appendices B1 to B8.

[0119] (Appendix B10) Each of the plurality of images further belongs to one of a plurality of middle classes obtained by classifying each of the plurality of classes into a plurality of middle classes, each middle class being a group of one or more of the subclasses; In the selection process, the at least one processor: Among the weak negative examples, an image that belongs to the same middle class as the target image belongs to is defined as a first weak negative example; Among the weak negative examples, an image belonging to a middle class different from the middle class to which the target image belongs is defined as a second weak negative example; In the re-sorting process, the at least one processor: re-sorting the plurality of images sorted by the sorting process into the positive example, the first weak negative example, the second weak negative example, and the strong negative example based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the first weak negative example, the similarity between the feature amount of the image of interest and the feature amount of the second weak negative example, and the similarity between the feature amount of the image of interest and the feature amount of the strong negative example; In the learning process, the at least one processor training the machine learning model so that a similarity between the feature of the image of interest and the feature of the first weak negative example is greater than a similarity between the feature of the image of interest and the feature of the second weak negative example; Study methods described in Appendix B2.

[0120] (Appendix B11) the subject included in the image is a specimen cell, The image is classified into classes depending on whether the specimen cells contained in the image are benign or malignant, and into subclasses depending on the type of specimen cells contained in the image. Study methods described in Appendix B2 or B10.

[0121] (Appendix B12) In the selection process, the at least one processor selects a data-augmented image obtained by data-augmenting the image of interest as a positive example. A study method described in one of Appendices B1 to B11.

[0122] (Appendix B13) The machine learning model is a first group of layers that receives an image as input and generates features of the image; a second layer group connected to the first layer group, which receives the image feature amount as an input and classifies the input image into the class or the subclass; Including, In the learning process, the at least one processor trains the machine learning model using a plurality of images and training labels corresponding to results of classifying each of the plurality of images into the class or the subclass. Study methods described in Appendix B2.

[0123] (Appendix B14) a classification process in which the at least one processor inputs an image including a subject into the machine learning model trained by the learning device according to any one of Supplementary Notes B1 to B13, and classifies the image into a class or a subclass; The classification method further comprises:

[0124] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0125] (Appendix C1) A program for causing a computer to function as a learning device, the program comprising: a selection means for selecting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images; a re-sorting means for re-sorting the plurality of images sorted by the sorting means into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning means for training a machine learning model using the plurality of images reselected by the reselection means so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A learning program to function as a

[0126] (Appendix C2) each of the plurality of images belongs to one of a plurality of classes and also belongs to one of a plurality of subclasses obtained by further classifying each of the plurality of classes; the labels indicate a class and a subclass to which each of the plurality of images belongs; The selecting means is selecting an image belonging to the subclass to which the target image belongs as a positive example; selecting, as weak negative examples, images that belong to a subclass different from the subclass to which the target image belongs and that belong to the class to which the target image belongs; selecting an image belonging to a class different from the class to which the target image belongs as a strong negative example; The study programme described in Appendix C1.

[0127] (Appendix C3) the re-selection means re-selects the plurality of images based on at least one of a distance between the target image and the weak negative examples and a distance between the target image and the strong negative examples in a predetermined feature space; A study program as described in Appendix C1 or C2.

[0128] (Appendix C4) The re-selection means re-selects the plurality of images based on a threshold determined by at least one of (i) a confidence interval and a maximum value of a distance between the target image and the weak negative example, and (ii) a confidence interval and a minimum value of a distance between the target image and the strong negative example. The study programme described in Appendix C3.

[0129] (Appendix C5) The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold value as the strong negative examples. The study programme described in Appendix C3.

[0130] (Appendix C6) The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as the weak negative examples. The study programme described in Appendix C3.

[0131] (Appendix C7) The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold, as examples not to be used in training the machine learning model. The study programme described in Appendix C3.

[0132] (Appendix C8) The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as examples not to be used in training the machine learning model. The study programme described in Appendix C3.

[0133] (Appendix C9) the feature amount referred to by the learning means is a feature vector, the similarity between the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example is based on an inner product of the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example; A study programme as set out in any one of Appendices C1 to C8.

[0134] (Appendix C10) Each of the plurality of images further belongs to one of a plurality of middle classes obtained by classifying each of the plurality of classes into a plurality of middle classes, each middle class being a group of one or more of the subclasses; The selecting means is Among the weak negative examples, an image that belongs to the same middle class as the target image belongs to is defined as a first weak negative example; Among the weak negative examples, an image belonging to a middle class different from the middle class to which the target image belongs is defined as a second weak negative example; The re-sorting means re-sorting the plurality of images selected by the sorting means into the positive example, the first weak negative example, the second weak negative example, and the strong negative example based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the first weak negative example, the similarity between the feature amount of the image of interest and the feature amount of the second weak negative example, and the similarity between the feature amount of the image of interest and the feature amount of the strong negative example; The learning means training the machine learning model so that a similarity between the feature of the image of interest and the feature of the first weak negative example is greater than a similarity between the feature of the image of interest and the feature of the second weak negative example; The study programme described in Appendix C2.

[0135] (Appendix C11) the subject included in the image is a specimen cell, The image is classified into classes depending on whether the specimen cells contained in the image are benign or malignant, and into subclasses depending on the type of specimen cells contained in the image. A study program as described in Appendix C2 or C10.

[0136] (Appendix C12) The selection means selects a data-augmented image obtained by data-augmenting the target image as a positive example. A study programme as set out in any one of Appendices C1 to C11.

[0137] (Appendix C13) The machine learning model is a first group of layers that receives an image as input and generates features of the image; a second layer group connected to the first layer group, which receives the image feature amount as an input and classifies the input image into the class or the subclass; Equipped with the learning means trains the machine learning model using a plurality of images and training labels corresponding to the results of classifying each of the plurality of images into the class or the subclass; The study programme described in Appendix C2.

[0138] (Appendix C14) The computer A class classification process in which an image including a subject is input to the machine learning model trained by the learning device according to any one of appendices C1 to C13, and the image is classified into a class or a subclass; A classification program that acts as a

[0139] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0140] (Appendix D1) at least one processor, a sorting process for sorting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label assigned to an image of interest included in the plurality of images and labels assigned to each of the plurality of images; a re-sorting process for re-sorting the images sorted by the sorting process into the positive examples, the weak negative examples, and the strong negative examples based on at least one of the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; a learning process for training a machine learning model using the plurality of images reselected by the reselection process so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A learning device that performs the following:

[0141] The learning device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0142] (Appendix D2) each of the plurality of images belongs to one of a plurality of classes and also belongs to one of a plurality of subclasses obtained by further classifying each of the plurality of classes; the labels indicate a class and a subclass to which each of the plurality of images belongs; In the selection process, the at least one processor: selecting an image belonging to the subclass to which the target image belongs as a positive example; selecting, as weak negative examples, images that belong to a subclass different from the subclass to which the target image belongs and that belong to the class to which the target image belongs; selecting an image belonging to a class different from the class to which the target image belongs as a strong negative example; 10. The learning device according to claim D1.

[0143] (Appendix D3) In the re-selection process, the at least one processor re-selects the plurality of images based on at least one of a distance between the target image and the weak negative examples and a distance between the target image and the strong negative examples in a predetermined feature space. A learning device according to appendix D1 or D2.

[0144] (Appendix D4) In the re-selection process, the at least one processor re-selects the plurality of images based on a threshold determined by at least one of (i) a confidence interval and a maximum value of a distance between the image of interest and the weak negative example, and (ii) a confidence interval and a minimum value of a distance between the image of interest and the strong negative example. 10. The learning device according to claim D3.

[0145] (Appendix D5) In the re-selection process, the at least one processor selects, from among the images selected as the weak negative examples by the selection process, images whose distance from the target image is greater than a threshold value as the strong negative examples. 10. The learning device according to claim D3.

[0146] (Appendix D6) In the re-selection process, the at least one processor selects, from among the images selected as the strong negative examples by the selection process, images whose distance from the target image is smaller than a threshold value as the weak negative examples. 10. The learning device according to claim D3.

[0147] (Appendix D7) In the re-selection process, the at least one processor selects, from among the images selected as the weak negative examples by the selection process, images whose distance from the target image is greater than a threshold, as examples not to be used in training the machine learning model. 10. The learning device according to claim D3.

[0148] (Appendix D8) In the re-selection process, the at least one processor selects, from among the images selected as the strong negative examples by the selection process, images whose distance from the target image is smaller than a threshold, as examples not to be used in training the machine learning model. 10. The learning device according to claim D3.

[0149] (Appendix D9) the feature amounts referred to in the learning process are feature vectors, the similarity between the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example is based on an inner product of the feature vector of the image of interest and each of the feature vectors of the positive example, the weak negative example, and the strong negative example; 10. The learning device according to any one of appendices D1 to D8.

[0150] (Appendix D10) Each of the plurality of images further belongs to one of a plurality of middle classes obtained by classifying each of the plurality of classes into a plurality of middle classes, each middle class being a group of one or more of the subclasses; In the selection process, the at least one processor: Among the weak negative examples, an image that belongs to the same middle class as the target image belongs to is defined as a first weak negative example; Among the weak negative examples, an image belonging to a middle class different from the middle class to which the target image belongs is defined as a second weak negative example; In the re-sorting process, the at least one processor: re-sorting the plurality of images sorted by the sorting process into the positive example, the first weak negative example, the second weak negative example, and the strong negative example based on at least one of the similarity between the feature amount of the image of interest and the feature amount of the first weak negative example, the similarity between the feature amount of the image of interest and the feature amount of the second weak negative example, and the similarity between the feature amount of the image of interest and the feature amount of the strong negative example; In the learning process, the at least one processor training the machine learning model so that a similarity between the feature of the image of interest and the feature of the first weak negative example is greater than a similarity between the feature of the image of interest and the feature of the second weak negative example; 10. The learning device of claim D2.

[0151] (Appendix D11) the subject included in the image is a specimen cell, The image is classified into classes depending on whether the specimen cells contained in the image are benign or malignant, and into subclasses depending on the type of specimen cells contained in the image. 10. The learning device according to claim D2 or D10.

[0152] (Appendix D12) In the selection process, the at least one processor selects a data-augmented image obtained by data-augmenting the image of interest as a positive example. 12. A learning device according to any one of appendices D1 to D11.

[0153] (Appendix D13) The machine learning model is a first group of layers that receives an image as input and generates features of the image; a second layer group connected to the first layer group, which receives the image feature amount as an input and classifies the input image into the class or the subclass; Equipped with In the learning process, the at least one processor trains the machine learning model using a plurality of images and training labels corresponding to results of classifying each of the plurality of images into the class or the subclass. 10. The learning device of claim D2.

[0154] (Appendix D14) The at least one processor: A class classification process in which an image including a subject is input to the machine learning model trained by the learning device according to any one of appendices D1 to D13, and the image is classified into a class or a subclass; A classification device that performs

[0155] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0156] (Appendix E1) A program for causing a computer to function as a learning device, the program comprising: a sorting process for sorting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label assigned to an image of interest included in the plurality of images and labels assigned to each of the plurality of images; a re-sorting process for re-sorting the images sorted by the sorting process into the positive examples, the weak negative examples, and the strong negative examples based on at least one of the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; a learning process for training a machine learning model using the plurality of images reselected by the reselection process so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A non-transitory recording medium on which a learning program for executing the above is recorded. [Explanation of symbols]

[0157] 1 Learning device 11 Sorting Department 12, 113A Re-sorting Department 13, 114A Learning Department 1A Information processing equipment 112A Sample Selection Unit 121A Classification Section LM1 machine learning model

Claims

1. a selection means for selecting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images; a re-sorting means for re-sorting the plurality of images sorted by the sorting means into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning means for training a machine learning model using the plurality of images reselected by the reselection means so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A learning device comprising:

2. each of the plurality of images belongs to one of a plurality of classes and also belongs to one of a plurality of subclasses obtained by further classifying each of the plurality of classes; the labels indicate a class and a subclass to which each of the plurality of images belongs; The selecting means is selecting an image belonging to the subclass to which the target image belongs as a positive example; selecting, as weak negative examples, images that belong to a subclass different from the subclass to which the target image belongs and that belong to the class to which the target image belongs; selecting an image belonging to a class different from the class to which the target image belongs as a strong negative example; The learning device according to claim 1 .

3. the re-selection means re-selects the plurality of images based on at least one of a distance between the target image and the weak negative examples and a distance between the target image and the strong negative examples in a predetermined feature space; The learning device according to claim 1 or 2.

4. The re-selection means re-selects the plurality of images based on a threshold determined by at least one of (i) a confidence interval and a maximum value of a distance between the target image and the weak negative example, and (ii) a confidence interval and a minimum value of a distance between the target image and the strong negative example. The learning device according to claim 3 .

5. The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold value as the strong negative examples. The learning device according to claim 3 .

6. The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as the weak negative examples. The learning device according to claim 3 .

7. The re-selection means selects, from among the images selected as the weak negative examples by the selection means, images whose distance from the target image is greater than a threshold value as examples not to be used in training the machine learning model. The learning device according to claim 3 .

8. The re-selection means selects, from among the images selected as the strong negative examples by the selection means, images whose distance from the target image is smaller than a threshold value as examples not to be used in training the machine learning model. The learning device according to claim 3 .

9. a sorting process in which at least one processor sorts the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label assigned to an image of interest included in the plurality of images and labels assigned to each of the plurality of images; a re-sorting process in which the at least one processor re-sorts the plurality of images sorted by the sorting process into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning process in which the at least one processor uses the plurality of images reselected by the reselection process to train a machine learning model such that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; Learning methods including.

10. A program for causing a computer to function as a learning device, the program comprising: a selection means for selecting the plurality of images into positive examples, weak negative examples, and strong negative examples based on a label attached to an image of interest included in the plurality of images and labels attached to each of the plurality of images; a re-sorting means for re-sorting the plurality of images sorted by the sorting means into the positive examples, the weak negative examples, and the strong negative examples based on at least one of a similarity between the feature amount of the image of interest and the feature amount of the weak negative examples and a similarity between the feature amount of the image of interest and the feature amount of the strong negative examples; a learning means for training a machine learning model using the plurality of images reselected by the reselection means so that (i) the similarity between the feature amounts of the image of interest and the feature amounts of the positive examples increases, (ii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples and the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples decreases, and (iii) the similarity between the feature amounts of the image of interest and the feature amounts of the weak negative examples becomes greater than the similarity between the feature amounts of the image of interest and the feature amounts of the strong negative examples; A learning program to function as a

Images