Computer system and learning data generation method

Abstract

To provide a computer system and a learning data generation method for generating learning data for efficiently learning a model that outputs a feature quantity vector of data to which a numerical value is given as a label.SOLUTION: A computer system that holds information of a model that calculates a feature amount vector of data to which a label that is a numerical value representing a feature of the data is assigned, calculates the feature amount vector by inputting the data to the model, and executes a process of selecting a negative example and a positive example using the feature amount vector a plurality of times, the computer system comprising: For each candidate data, an index representing the difference between the query and the label of the candidate data is calculated, the candidate data having a large index is selected as a negative example, and the candidate data having a small index is selected as a positive example.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for generating training data. [Background technology]

[0002] One well-known information processing technology for handling images is similar image search, which converts an image into an image feature vector that represents its content, evaluates the similarity between images, and searches for images similar to a given query image. By searching for similar images, it is possible to search for information associated with the image.

[0003] Various feature extraction methods have been proposed to construct image feature vectors. Recently, extraction methods using deep neural networks have become popular.

[0004] One of the learning methods for deep neural networks is the triplet network. In a triplet network, a model (e.g., a neural network) is trained to output feature vectors that have a small distance between data with the same label and a large distance between data with different labels. Contrastive learning is also known as a similar learning method.

[0005] Non-Patent Document 1 describes a technology for estimating the calorie value of a meal by searching for images of the meal to which calorie values ​​are assigned as labels. In this document, a model is constructed that outputs a feature vector using a triplet network learning method that uses food categories as labels. While the technology in this example could be used in services that aim to improve dietary habits, there is a wide range of needs for technology that estimates numerical values ​​from images. For example, there is also an application example in estimating expected yields from images of agricultural crops.

[0006] As a method for constructing feature vectors, Non-Patent Document 2 describes triplet network learning that performs a similar vector search during the learning process. Non-Patent Document 2 describes triplet network learning that performs a similar vector search using the feature vector at that time at each epoch in the learning process and selects the optimal triplet for learning. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Takumi Aishita and Keiji Yanai, Comparison of Regression-Based and Search-Based Methods for Food Image Calorie Estimation, Information Processing Society of Japan, Computer Vision and Image Media (CVIM), 2017 [Non-patent document 2] Yasumitsu Ikeura, Koichi Okamoto, Ryohei Kashima, Atsushi Hiroike, Multimodal Deep Learning Platform for IoT Data, Hitachi Review, 102-3, 119-123 (2020) Summary of the Invention [Problem to be solved by the invention]

[0008] The learning method described in Non-Patent Document 1 requires that information corresponding to food categories be attached to images in addition to information corresponding to calorie values. Therefore, in the case of images that are accompanied only by information that cannot be simply classified into groups such as categories, the learning method described in Non-Patent Document 1 cannot form triplets.

[0009] On the other hand, it is possible to reduce the calorie values ​​to several ranges and treat each range as a label. However, because each range contains a variety of food images, a deeper model is required for highly accurate inference. Furthermore, training such a model requires a large amount of training data. Therefore, it is difficult to realize training from the perspective of computer resources and preparation of training data.

[0010] The above two issues are common in learning methods that reduce the estimation of numerical information to classification learning into categories and construct feature vectors. For example, contrastive learning based on contrastive loss is one such learning method. [Means for solving the problem]

[0011] A representative example of the invention disclosed in the present application is as follows: That is, a computer system having at least one computer holds information about a model for calculating a feature vector of data to which at least one label is assigned, the at least one label being a numerical value representing a feature of the data, the computer system acquires a plurality of pieces of data, inputs the plurality of pieces of data into the model to calculate the feature vector, selects one piece of data as a query, and executes a process multiple times to select negative examples and positive examples anchored by the query using the feature vector, calculates a distance between the query and the feature vector of the piece of data, determines a similarity ranking of the plurality of pieces of data to the query based on the distance, selects a predetermined number of pieces of data as candidate data in descending order of similarity ranking, calculates an index representing a difference between the query and the at least one label of the candidate data for each of the selected pieces of candidate data, and selects the candidate data with a larger index as the negative example and the candidate data with a smaller index as the positive example. [Effects of the Invention]

[0012] According to the present invention, it is possible to generate training data (positive examples and negative examples) for efficiently training a model that calculates feature vectors of data such as images, to which numerical values ​​are assigned as labels. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to a first embodiment. [Figure 3] 4 is a flowchart illustrating an example of a learning process executed by the learning device according to the first embodiment. [Figure 4] 10 is a flowchart illustrating an example of triplet set generation processing executed by the learning device of the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of triplet generation processing executed by the learning device according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating a specific example of triplet generation processing according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Examples of the present invention will be described below with reference to the drawings. However, the present invention should not be construed as being limited to the description of the following examples. Those skilled in the art will readily understand that the specific configuration can be modified without departing from the spirit or scope of the present invention. In the configurations of the invention described below, identical or similar components or functions are designated by the same reference numerals, and redundant description will be omitted. The designations "first," "second," "third," and the like in this specification are used to identify components and do not necessarily limit the number or order. The position, size, shape, and range of each component shown in the drawings may not represent the actual position, size, shape, and range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, and range, etc., disclosed in the drawings, etc. [Example]

[0015] In this embodiment, negative and positive examples for anchors are selected and a triplet network, which is one of the learning methods using these, is used as an example. Note that the present invention can also be applied to other learning methods that use negative and positive examples for learning, such as contrastive learning.

[0016] Fig. 1 is a diagram illustrating an example of a functional configuration of a learning device according to a first embodiment, and Fig. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to a first embodiment.

[0017] In the first embodiment, a system that handles food images to which calorie values ​​are assigned is assumed.

[0018] The learning device 100 uses images with calorie values ​​to learn a model that outputs feature vectors used for image classification. As shown in FIG. 2, the learning device 100 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. The hardware elements are connected to each other via a bus. The learning device 100 may have input devices such as a keyboard, a mouse, and a touch panel, and may also have output devices such as a display and a printer.

[0019] The processor 201 executes a program stored in the main memory device 202. The processor 201 executes processing in accordance with the program, thereby operating as a functional unit (module) that realizes a specific function. In the following description, when a processing is described using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0020] The main memory device 202 is a storage device such as a DRAM (Dynamic Random Access Memory) and stores programs executed by the processor 201 and data used by the programs. The main memory device 202 is also used as a work area. The secondary memory device 203 is a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and permanently stores data.

[0021] The programs and data stored in the main memory device 202 may be stored in the secondary memory device 203. In this case, the processor 201 reads the programs and data from the secondary memory device 203 and loads them into the main memory device 202.

[0022] The learning device 100 includes an inference unit 101, a similar vector search unit 102, a triplet generation unit 103, a model update unit 104, and a model storage unit 105. The learning device 100 also stores a dataset 110 and model information 120.

[0023] The data set 110 is made up of a plurality of training data 111. The training data 111 is an image to which a calorie value is assigned.

[0024] The model information 120 is information for managing a model that outputs a feature vector of an image. The model is, for example, a deep neural network. The model information 120 stores information about the structure and weights of the network. The weights of the model at the start of learning may be set randomly, but are assumed to be set to arbitrary values ​​in advance to improve the efficiency of learning.

[0025] The dataset 110 and the model information 120 may be set in advance in the learning device 100, obtained from an external source, or input by the user.

[0026] The inference unit 101 inputs images included in the training data 111 into a model defined by the model information 120, and outputs a feature vector.

[0027] The similar vector search unit 102 searches for similar feature vectors.

[0028] The triplet generation unit 103 generates triplets by selecting a set of images (learning data 111) that are anchors, positive examples, and negative examples based on the search results of similar vectors.

[0029] The model update unit 104 executes model learning processing using a plurality of triplets to update the model. Note that the present invention may be applied to any learning method using triplets.

[0030] The model storage unit 105 reflects the learning result in the model information 120. Note that the model storage unit 105 may output the learning result to an external storage device or the like as necessary.

[0031] FIG. 3 is a flowchart illustrating an example of the learning process executed by the learning device 100 according to the first embodiment.

[0032] The inference unit 101 acquires model information from the model information 120, and also acquires training data 111 from the data set 110 (step S301).

[0033] The inference unit 101 uses the model to execute an inference process for outputting a feature vector of the training data 111 (step S302).

[0034] The similar vector search unit 102 executes a clustering process of the feature vectors (step S303). In the clustering process, clusters of feature vectors are generated in the feature space. This process is a preprocessing for speeding up the search for similar feature vectors. Similar vector search using clustering is described in Non-Patent Document 2. Note that the clustering process does not necessarily have to be executed.

[0035] The triplet generation unit 103 executes triplet set generation processing in cooperation with the similar vector search unit 102 (step S304). The triplet set generation processing will be described in detail later.

[0036] The triplet generation unit 103 selects triplets to be used for learning from the triplet set generated in the triplet set generation process (step S305). In this embodiment, a predetermined number of triplets are selected in descending order of the loss function value of the triplet. The triplet loss function is defined as follows:

[0037]

number

[0038] Here, q is the anchor, s is the subscript representing a positive example, and r is the subscript representing a negative example. x_q, x_s, and x_r represent feature vectors, and y_q, y_s, and y_r represent calorie values. b is a parameter that defines the learning process; if the difference in squared distance between feature vectors is smaller than b, the value of the max function will be 0.

[0039] The triplet generation unit 103 outputs a triplet array having the selected triplets as elements to the model update unit 104. The array stores the triplets in an ordered manner.

[0040] The model update unit 104 updates the model using the selected triplet (step S306). In this embodiment, a function obtained by adding the loss functions of the selected triplet is used as the objective function, and a learning process is executed to minimize the objective function.

[0041] The processing from step S302 to step S306 is defined as one learning cycle (one epoch).

[0042] The learning device 100 determines whether or not a learning termination condition is met (step S307), for example, when the number of epochs is greater than a threshold value.

[0043] If the learning termination condition is not met, the learning device 100 returns to step S302 and executes the same process. If the learning termination condition is met, the learning device 100 ends the learning process.

[0044] FIG. 4 is a flowchart illustrating an example of triplet set generation processing executed by the learning device 100 according to the first embodiment.

[0045] The similar vector search unit 102 initializes a triplet set (step S401).

[0046] The similar vector search unit 102 selects one image to be used as a query from the training data 111 (step S402). The query may be selected randomly, or a selection method may be specified in advance.

[0047] The similar vector search unit 102 searches for images with feature vectors similar to the query feature vector (step S403). For example, the similar vector search unit 102 uses the feature vectors to calculate the distance between images in feature space, and outputs images with a distance smaller than a threshold as similar images. At this time, the similar vector search sorts the similar images in descending order of similarity and assigns numbers (ranks) starting from 1.

[0048] The similar vector search unit 102 calls the triplet generation unit 103. The triplet generation unit 103 executes triplet generation processing based on the search results (step S404), and after the processing is completed, outputs a response to the similar vector search unit 102. The triplet generation processing will be described in detail later.

[0049] The similar vector search unit 102 determines whether to end the triplet set generation process (step S405). For example, if an upper limit is set on the number of images selected as a query due to requirements such as restrictions on the amount of calculation, the similar vector search unit 102 ends the triplet set generation process if the number of times the loop process from step S402 to step S404 is performed is greater than a threshold. Note that the loop process may be performed for all of the training data 111.

[0050] Fig. 5 is a flowchart illustrating an example of triplet generation processing executed by the learning device 100 according to the first embodiment. Fig. 6 is a diagram illustrating a specific example of triplet generation processing according to the first embodiment.

[0051] The triplet generating unit 103 selects N similar images in descending order of similarity to the query (step S501), where N is an integer greater than 1 and is set in advance.

[0052] The triplet generation unit 103 calculates the difference D_i between the calorie value of the query and the calorie value of the similar image for each similar image (step S502). The subscript i is the ranking given to the similar image and is a value ranging from 1 to N.

[0053] The triplet generating unit 103 identifies the rank r of the similar image that maximizes the difference D_i within the rank range from 1 to M (step S503), where M is an integer smaller than N and is set in advance.

[0054] The triplet generating unit 103 identifies the rank s of the similar image that minimizes the difference D_i within the rank range from r+1 to N (step S504).

[0055] The triplet generation unit 103 determines whether the difference D_s is smaller than the difference D_r (step S505).

[0056] If the difference D_s is equal to or greater than the difference D_r, the triplet generation unit 103 ends the triplet generation process.

[0057] If the difference D_s is smaller than the difference D_r, the triplet generation unit 103 generates a triplet in which the training data 111 corresponding to the query is the anchor, the training data 111 corresponding to the s-th similar image is the positive case, and the training data 111 corresponding to the r-th similar image is the negative case (step S506), and adds the generated triplet to the triplet set (step S507). Thereafter, the triplet generation unit 103 ends the triplet generation process.

[0058] FIG. 6 shows the relationship between the difference D_i between the query and similar images. The horizontal axis represents the ranking, and the vertical axis represents the difference D_i. Note that since the query itself is the most similar to the query, the origin of the horizontal axis is the query. The reference line 600 represents the origin of the difference D_i. The numerical value of the image represents the ranking.

[0059] In image-based calorie inference, it is desirable that images with small differences D_i are ranked high in similarity, and that no images with large differences D_i exist. Therefore, in this embodiment, triplets are generated using images that contradict the above-mentioned situation.

[0060] In the first embodiment, by selecting images that make up a triplet from among the top images in a similarity search, it is possible to efficiently train a model without preparing a large amount of image data.

[0061] In this embodiment, the training data 111 with the largest difference D_i among the top M cases is selected as a candidate for a negative case, but multiple triplets can also be generated using a similar procedure by selecting a certain number of triplets with the largest difference D_i.

[0062] When inferring calorie values ​​from a new image, the learning device 100 calculates a feature vector using the trained model and performs a similar vector search on a set of images with calorie values, using the new image as a query. Furthermore, the learning device 100 outputs a statistical value, such as the average calorie value of the top K similar images, as an estimated calorie value. K is an integer greater than 1 and is set in advance.

[0063] When the present invention is applied to contrastive learning, the triplet set generation process becomes a process of generating pairs of positive and negative examples. Furthermore, the triplet generation process becomes a process of selecting positive and negative examples. In the case of targeted learning, each positive and negative example is used as a piece of learning data.

[0064] In the examples, an image to which a calorie value is assigned has been described as an example, but the present invention is not limited to this. An image to which multiple numerical values ​​are assigned as labels may also be used. In this case, the squared distance of the vector with the labels as elements can be used as the difference D_i.

[0065] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0066] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0067] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).

[0068] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0069] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0070] 100 Learning Device 101 Reasoning part 102 Similar Vector Search Unit 103 triplet generator 104 Model Update Section 105 Model Storage Unit 110 datasets 111 Training Data 120 Model Information 201 processor 202 Main storage 203 Secondary storage device 204 Network Interface

Claims

1. A computer system having at least one computer, The method stores information about a model that calculates a feature vector of data to which at least one label is assigned, The at least one label is a numerical value representing a feature of the data, The computer system acquiring a plurality of said data; inputting a plurality of the data into the model to calculate the feature vector; selecting one of the data as a query, and using the feature vector, executing a process of selecting negative examples and positive examples with the query as an anchor multiple times; In the process, Calculating the distance between the feature vectors of the query and the data; determining a similarity ranking of the plurality of data items with respect to the query based on the distance; selecting a predetermined number of the data items as candidate data items in descending order of similarity; calculating an index representing a difference between the query and the at least one label of the candidate data for each of the selected plurality of candidate data; The computer system is characterized in that the candidate data with a large index is selected as the negative example, and the candidate data with a small index is selected as the positive example.

2. 2. The computer system of claim 1, selecting, as the negative example, the candidate data with the largest index from among a plurality of candidate data included in a group generated by selecting a predetermined number of the candidate data in descending order of similarity rank; A computer system characterized in that the candidate data with the smallest index is selected as the positive case from among multiple candidate data whose similarity ranking is lower than the similarity ranking of the candidate data selected as the negative case.

3. 3. The computer system according to claim 2, the data is an image, A computer system that executes a learning process that updates the model using a plurality of the negative examples and a plurality of the positive examples.

4. 4. The computer system according to claim 3, generating a triplet consisting of the query, the negative examples, and the positive examples; A computer system that executes the learning process using a plurality of the triplets.

5. A method for generating training data executed by a computer system having at least one computer, comprising: the computer system holds information about a model for calculating a feature vector of data to which at least one label is assigned; The at least one label is a numerical value representing a feature of the data, The method for generating training data includes: a first step in which the computer system acquires a plurality of the data; a second step in which the computer system inputs a plurality of the data into the model and calculates the feature vector; a third step in which the computer system selects one of the data as a query, and executes a process of selecting negative examples and positive examples using the feature vector with the query as an anchor multiple times; The third step includes: a fourth step in which the computer system calculates a distance between the feature vectors of the query and the data; a fifth step in which the computer system determines a similarity ranking of the plurality of data with respect to the query based on the distance; a sixth step in which the computer system selects a predetermined number of the data items as candidate data items in descending order of similarity; a seventh step in which the computer system calculates, for each of the selected plurality of candidate data, an index representing a difference between the query and the at least one label of the candidate data; an eighth step in which the computer system selects the candidate data with a large index as the negative example and selects the candidate data with a small index as the positive example.

6. The training data generation method according to claim 5, The eighth step includes: a step in which the computer system selects, as the negative example, the candidate data with the largest index from among a plurality of the candidate data included in a group generated by selecting a predetermined number of the candidate data in descending order of similarity rank; and selecting, by the computer system, as the positive example, the candidate data having the smallest index from among a plurality of candidate data whose similarity ranking is lower than the similarity ranking of the candidate data selected as the negative example.

7. The training data generation method according to claim 6, the data is an image, A method for generating training data, comprising a ninth step in which the computer system executes a training process for updating the model using a plurality of the negative examples and a plurality of the positive examples.

8. The training data generation method according to claim 7, The ninth step includes: generating a triplet consisting of the query, the negative example, and the positive example by the computer system; a step in which the computer system executes the learning process using a plurality of the triplets.

Images