Information processing apparatus, method, and program

Abstract

To reduce calculation cost during training.SOLUTION: An information processing apparatus includes a feature extraction unit, a prediction unit, a negative example regularization unit, and a training unit. The feature extraction unit extracts a plurality of features from multiple pieces of training data by using a machine learning model. The prediction unit generates a prediction result relating to a task, from the training data and teaching data corresponding to the training data. The negative example regularization unit calculates a similarity between features with respect to the plurality of features. The training unit updates a parameter of the machine learning model, based on the prediction result and the similarity, in such a manner that the features become farther from each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] In machine learning, the human cost of generating training data can be reduced by training a model using self-supervised learning, which uses automatically added training signals instead of manually prepared training data. One such self-supervised learning method is contrastive learning. In contrastive learning, data pairs are generated from one piece of data by data augmentation, and the model is trained to bring the pairs closer to each other as positive examples and to move pairs other than the positive examples (negative example pairs) farther apart. However, there is a problem in that the amount of calculation is large because it is necessary to input images related to the positive example pairs and images related to the negative example pairs into the encoder respectively. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Pat. No. 1,354,778 [Patent Document 2] U.S. Pat. No. 1,138,6302 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an information processing device, method, and program that can reduce the calculation cost during training. [Means for solving the problem]

[0005] The information processing device according to the present embodiment includes a feature extraction unit, a prediction unit, a negative example regularization unit, and a training unit. The feature extraction unit extracts a plurality of feature amounts from a plurality of training data using a machine learning model. The prediction unit generates a prediction result related to a task from the training data and teacher data corresponding to the training data. The negative example regularization unit calculates a similarity between the plurality of feature amounts. The training unit updates parameters of the machine learning model based on the prediction result and the similarity so that the feature amounts move away from each other. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an information processing apparatus according to an embodiment of the present invention. [Diagram 2] 5 is a flowchart showing a first operation example of the information processing device according to the embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a machine learning model used in a first operation example of the information processing device according to the embodiment. [Figure 4] FIG. 11 is a diagram showing another example of a machine learning model used in the first operation example of the information processing device according to the embodiment. [Diagram 5] 10 is a flowchart showing a second operation example of the information processing device according to the embodiment. [Figure 6] FIG. 13 is a diagram showing a second example of a network structure of a machine learning model according to this embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a hardware configuration of an information processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an information processing device, method, and program according to the present embodiment will be described in detail with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and duplicated descriptions will be omitted as appropriate.

[0008] The information processing device according to this embodiment will be described with reference to the block diagram of FIG. The information processing device 10 according to the first embodiment includes a storage unit 101, an acquisition unit 102, a generation unit 103, a feature extraction unit 104, a negative example regularization unit 105, a prediction unit 106, and a training unit 107.

[0009] The storage unit 101 stores a machine learning model, training data, teacher data, and the like. The machine learning model is, for example, an autoencoder that combines models called an encoder and a decoder. Specifically, the encoder is assumed to be a Vision Transformer (ViT). Note that the present invention is not limited to ViT, and any model can be applied as long as it is a network model used in feature extraction or dimensional compression, such as a deep network model including a convolutional neural network (CNN) such as ResNet.

[0010] It is assumed that the decoder uses a network model composed of a Transformer block. The decoder is not limited to the Transformer block, and any model can be applied as long as it is a network model capable of generating restored data from features extracted by the encoder, such as MLP (Multi-Layer Perceptron) or convolutional neural network. The training data is data for learning a learning model described later. The teacher data is correct answer data for the training data. In this embodiment, the training data is assumed to be an image, but is not limited to this. For example, it may be one-dimensional time series data such as control signal data, voice data, natural language data, music data, sensor values, or two-dimensional or higher data other than images such as symbol data, log data, video data, table data, and transaction data.

[0011] The acquiring unit 102 acquires training data and teacher data. The generating unit 103 generates a plurality of partial training data from one training data by extracting a portion of the training data, and also performs data augmentation processing on the plurality of training data. The feature extraction unit 104 extracts a plurality of feature quantities from a plurality of training data by using a machine learning model.

[0012] The negative example regularization unit 105 calculates the similarity between a plurality of feature amounts. The prediction unit 106 generates a prediction result for a task from the training data and teacher data corresponding to the training data. The training unit 107 updates the parameters of the machine learning model based on the prediction result and the similarity so that the features move away from each other. The parameters of the machine learning model include, for example, the weights and biases of the network.

[0013] Next, a first operation example of the information processing device 10 according to the present embodiment will be described with reference to the flowchart in Fig. 2. Here, it is assumed that an autoencoder is used as the machine learning model.

[0014] In step SA1 , the acquisition unit 102 acquires the training data and teacher data from the storage unit 101 . In step SA2, the feature extraction unit 104 extracts features from the training data. For example, the feature extraction unit 104 may input the training data to an encoder to extract features related to the training data.

[0015] In step SA3, the prediction unit 106 uses the features and teacher data to generate a prediction result for the task. For example, in the case of a classification task, the features are input to a decoder, and an output from the decoder regarding classification is generated as a prediction result. Furthermore, a first loss L task Calculate.

[0016]

number

[0017] Here, N is the number of data, and y jis the prediction result for the jth training data, and x j is the jth training data.

[0018] In step SA4, the negative example regularization unit 105 calculates a loss for a pair of features, i.e., a second loss for the similarity between the features, for the multiple features calculated in step SA3. Specifically, the second loss L reg It is sufficient to calculate

[0019]

number

[0020] Here, Z i are the features of each data. τ is a temperature parameter and may be set to any value. The loss function shown in formula (2) is a function whose value decreases as the similarity between features decreases. In other words, the loss function shown in formula (2) is a loss function for updating weights so that the features move away from each other. Note that the loss function used by the negative example regularization unit 105 is not limited to formula (2), and may be, for example, formula (3).

[0021]

number

[0022] where I is the identity matrix, z is the feature matrix, and z T is the transposed matrix of the features. Also, the symbol shown as a cross in a circle is a tensor product.

[0023] Note that the formula (4) may be used as the loss function of the negative example regularization unit 105. Of course, the loss function is not limited to these formulas, and any loss function may be adopted as long as it can train a model so that features move away from each other.

[0024]

number

[0025] In step SA5, the training unit 107 task and the second Ross L reg The third loss L, which is the loss of the entire model, is calculated using the above formula (5). Specifically, the third loss L may be calculated using the following formula (5), for example.

[0026]

number

[0027] α is an arbitrary real number, and is the second loss L calculated by the negative example regularization unit 105. reg This is a parameter for weighting.

[0028] In step SA6, the training unit 104 determines whether the training of the machine learning model has been completed. For example, the training may be determined to have been completed when the loss value of the loss function of equation (5) is equal to or less than a threshold. Alternatively, the training may be determined to have been completed when the decrease in the loss value has converged. Furthermore, the training may be determined to have been completed when a predetermined number of epochs have been processed. If the training has been completed, the processing is terminated, and if the training has not been completed, the processing proceeds to step SA7.

[0029] In step SA7, the training unit 104 updates the parameters of the machine learning model (encoder here) by, for example, gradient descent and backpropagation so as to minimize the loss value, specifically, updates parameters such as weights and biases of the neural network related to the encoder. After updating the parameters, the process returns to step SA1, and training of the learning model is continued using new training data. When training is finally completed, training of the machine learning model ends, and a trained model is generated. The trained trained model may be stored in, for example, the storage unit 101 of the information processing device 10.

[0030] Next, an example of a machine learning model used in the first operation example of the information processing device 10 according to the present embodiment will be described with reference to FIG. The machine learning model 30 shown in FIG. 3 is an example of an autoencoder, and includes an encoder 31 and a decoder 32. In the autoencoder shown in FIG. 3, a prediction task of predicting an original image is assumed. The encoder 31 and the decoder 32 may have a symmetrical structure or a non-symmetrical structure. In addition, the model size of the machine learning model of the decoder 32 may be designed to be smaller than the model size of the encoder 31. This can reduce the amount of calculation. In the following, when no particular distinction is made, they will be referred to as an input image 33, a feature amount 34, and an output image 35.

[0031] In the example of FIG. 3, an input image 33 is input to the encoder 31, and features are extracted in the encoder 31. Here, an input image 33-1 of a cat and an input image 33-2 of an elephant are input as the input image 33. In the encoder 31, the input image 33-1 and the input image 33-2 are each subjected to a convolution process, and a feature 34-1 related to the input image 33-1 and a feature 34-2 related to the input image 33-2 are each extracted. Here, the second loss between the feature 34-1 and the feature 34-2 is calculated by the process of the negative example regularization unit 105, for example, by any of the loss functions of the above-mentioned formulas (2) to (4), which can be trained so that the features move away from each other. That is, in the information processing device 10 according to this embodiment, instead of training to bring positive examples closer and negative examples farther away as in contrastive learning, training is performed to move all features away from each other. This eliminates the need to prepare positive examples, and reduces costs.

[0032] Meanwhile, the decoder 32 receives the features 34-1 and 34-2 output from the encoder 31. The decoder 32 outputs output images 35-1 and 35-2 that reproduce the input images 33-1 and 33-2. The prediction unit 106 calculates a first loss by a loss function that can be trained to reduce the difference between the output image 35 and the input image 33, for example, the loss function of the above-mentioned equation (1).

[0033] In addition, the classification task may be solved using training data. Another example of the machine learning model used in the first operation example of the information processing device 10 according to the present embodiment will be described with reference to FIG. In the machine learning model 40 shown in Fig. 4, a classifier 41 that performs class classification is connected to the encoder 31 instead of the decoder shown in Fig. 3. For example, the prediction unit 106 outputs a probability value of the classification task as a predicted value from the classifier 41 instead of the output image 35. Also, the model 40 may be trained so that the classification task can be solved by a loss function related to the cross-entropy error between the predicted value and the teacher data instead of the first loss. In this case, the prediction unit 106 may use an MLP consisting of one or two layers as the classifier 41 instead of the decoder.

[0034] Next, a second operation example of the information processing device 10 according to the present embodiment will be described with reference to the flowchart of FIG. Step SA1 and steps SA3 to SA7 are the same as those in FIG. 2, and therefore the description thereof will be omitted.

[0035] In step SB1, the generating unit 103 generates a plurality of partial training data by cutting out a part of the training data. For example, partial images generated by cropping a plurality of partial regions, such as the upper right part and the lower left part, of one image data may be used as the partial training data. The generating unit 103 may perform data augmentation processing on the training data. The data augmentation processing can generate a plurality of data based on one image, including not only cropped images but also image division (patching), inversion, rotation, noise addition, and combinations of these.

[0036] In step SB2, the feature extraction unit 104 extracts features from the partial training data. In this way, by using partial training data, the amount of calculation in the encoder can be reduced.

[0037] Next, an example of a machine learning model used in a second operation example of the information processing device 10 according to the present embodiment will be described with reference to FIG. 6, a model using a Vision Transformer (ViT) is used as the encoder 51. The ViT has a structure in which a plurality of Transformer Encoder blocks are stacked.

[0038] In ViT, first, the input image 33 is divided into patch images by a patch embedding process 61-1. Next, position embedding process 61-2 embeds position information indicating the position of the patch image in the whole image and a learnable class token into the patch image. A detailed description of the class token 62 is omitted since it is assumed that the token is generally used in ViT. Note that the patch embedding process 61-1 and the position embedding process 61-2 may be performed as a process separate from the encoder, i.e., as a pre-processing, without being incorporated into the encoder 61.

[0039] Thereafter, the patch image with the embedded position information is processed by a multi-layer Transformer Encoder block 61-3 to extract the feature amount 34. Note that the Transformer Encoder block 61-3 itself is a typical process in ViT, and therefore a detailed description thereof will be omitted here.

[0040] As in the case of FIG. 3, the feature 34 output from the encoder 61 is trained by the negative example regularization unit 105 using a trainable loss function that moves all feature values ​​away from each other. As a result, the second loss L reg As for the decoder 63, as in the case of FIG. 3, a loss function that reduces the difference between the output image 35 and the input image 33, or a loss function that reduces the difference between the output image 35 and the image before the input image 33 is augmented, is used to calculate the first loss L task can be calculated.

[0041] After that, the machine learning model of FIG. 6 may be trained using, for example, a loss function for the loss L of the entire model shown in equation (5).

[0042] Next, an example of a hardware configuration of the information processing device 10 according to the above embodiment is shown in a block diagram in FIG. The information processing device 10 includes a CPU (Central Processing Unit) 71, a RAM (Random Access Memory) 72, a ROM (Read Only Memory) 73, a storage 74, a display device 75, an input device 76, and a communication device 77, each of which is connected by a bus.

[0043] The CPU 71 is a processor that executes arithmetic processing and control processing according to a program. The CPU 71 executes the processing of each part of the information processing device 10 described above in cooperation with the programs stored in the ROM 73 and the storage 74, etc., using a predetermined area of ​​the RAM 72 as a working area. Each process of the information processing device 10 may be executed by one processor, or may be executed in a distributed manner by multiple processors.

[0044] The RAM 72 is a memory such as a Synchronous Dynamic Random Access Memory (SDRAM), and functions as a work area for the CPU 71. The ROM 73 is a memory that stores programs and various information in a non-rewritable manner.

[0045] The storage 74 is a device that writes and reads data to a magnetic recording medium such as a hard disk drive (HDD), a semiconductor storage medium such as a flash memory, a magnetically recordable storage medium such as a HDD, an optically recordable storage medium, etc. The storage 74 writes and reads data to the storage medium in response to control from the CPU 71.

[0046] The display device 75 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 75 displays various information based on a display signal from the CPU 71.

[0047] The input device 76 is an input device such as a mouse, a keyboard, etc. The input device 76 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 71.

[0048] The communication device 77 communicates with external devices via a network under the control of the CPU 71 .

[0049] According to the present embodiment described above, a prediction result for a task is generated from training data and teacher data, the similarity between features of the training data is calculated, a loss function is set based on the prediction result and the similarity so that the features move away from each other, and the parameters of the machine learning model are updated based on the loss value of the loss function. This eliminates the need to generate positive examples and input pairs of training data and positive examples and pairs of training data and negative examples to the encoder for calculations, as in general contrastive learning. Also, it is not necessary to generate mask images (images or patch images in which a part of the input image is masked) and input them to the encoder, as in Masked Auto Encoder (MAE). This reduces the computational cost of training the machine learning model. Furthermore, by making the model size of the decoder smaller than the model size of the encoder in the machine learning model, the amount of computation can be further reduced.

[0050] Here, we assume that the model size of the decoder is sufficiently small compared to the encoder, and the computational complexity of the decoder can be ignored. A neural network is used as the machine learning model, and the backpropagation method, which is a common training method for the network, is used. In this case, if the computational complexity of the encoder's forward propagation per data is defined as C, the computational complexity of the backpropagation and the computational complexity of the differential calculations in the backpropagation have a total computational cost of 2C. Furthermore, in contrastive learning, since both pairs of positive examples and pairs of negative examples are input to the encoder, the computational complexity of the forward propagation is 2C, and as a result, a total computational complexity of approximately 4C is required per training data.

[0051] On the other hand, the computational cost during training by the information processing device 10 according to this embodiment is about 3C because no calculations are required for positive example pairs. Therefore, compared to contrastive learning, the amount of calculations per training data can be reduced by 25%.

[0052] The instructions shown in the processing procedure shown in the above-mentioned embodiment can be executed based on a program, which is software. A general-purpose computer system can store this program in advance and obtain the same effect as the effect of the control operation of the above-mentioned information processing device by reading this program. The instructions described in the above-mentioned embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray (registered trademark) Disc, etc.), a semiconductor memory, or a recording medium similar thereto. The recording medium may be in any storage format as long as it is readable by a computer or an embedded system. The computer can realize the same operation as the control of the information processing device of the above-mentioned embodiment by reading the program from this recording medium and executing the instructions described in the program based on this program with a CPU. Of course, when the computer acquires or reads the program, it may acquire or read it through a network. In addition, an OS (operating system), database management software, network, or other MW (middleware) running on a computer may execute some of the processes required to realize this embodiment based on instructions from a program installed on the computer or embedded system from a recording medium. Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like. Furthermore, the number of recording media is not limited to one, and cases in which the processing in this embodiment is executed from multiple media are also included in the recording media in this embodiment, and the media may have any configuration.

[0053] The computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored in a recording medium, and may be configured as any one of a device such as a personal computer or a microcomputer, or a system in which multiple devices are connected to a network. In addition, the computer in this embodiment is not limited to a personal computer but also includes an arithmetic processing device, a microcomputer, etc. included in information processing equipment, and is a general term for equipment or devices that can realize the functions in this embodiment by a program.

[0054] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0055] 10...information processing device, 30,40...machine learning model, 31,61...encoder, 32,63...decoder, 33-1,33-2...input image, 34-1,34-2...feature, 35-1,35-2...output image, 41...classifier, 61-1...patch embedding process, 61-2...position embedding process, 61-3...Transformer Encoder block, 62...class token, 71...CPU, 72...RAM, 73...ROM, 74...storage, 75...display device, 76...input device, 77...communication device, 101...storage unit, 102...acquisition unit, 103...generation unit, 104...feature extraction unit, 105...negative example regularization unit, 106...prediction unit, 107...training unit.

Claims

1. a feature extraction unit that extracts a plurality of feature quantities from a plurality of training data using a machine learning model; a prediction unit that generates a prediction result regarding a task from the training data and teacher data corresponding to the training data; a negative example regularization unit that calculates a similarity between the plurality of feature amounts; a training unit that updates parameters of the machine learning model based on the prediction result and the similarity so that the feature quantities move away from each other; An information processing device comprising:

2. A generating unit that generates a plurality of partial training data from one training data, The information processing apparatus according to claim 1 , wherein the feature extraction unit extracts the feature amounts by using the plurality of partial training data.

3. The information processing device according to claim 1 , wherein the model used by the prediction unit has a smaller model size than that of the feature extraction unit.

4. The information processing apparatus according to claim 1 , wherein the task is a process of predicting the training data to be input to the feature extraction unit.

5. A generator that performs a data augmentation process on the plurality of training data, The information processing apparatus according to claim 1 , wherein the task is a process of predicting the training data before the data augmentation process is executed.

6. The information processing device according to claim 1 , wherein the training unit updates parameters of the machine learning model so as to minimize a loss value calculated by a loss function that has a smaller value as the similarity between the feature amounts decreases.

7. A machine learning model is used to extract multiple features from multiple training data, generating a prediction result for a task from the training data and teacher data corresponding to the training data; Calculating similarities between the plurality of feature amounts; updating parameters of the machine learning model based on the prediction result and the similarity so that the feature quantities move away from each other; Information processing methods.

8. Computer, A feature extraction means for extracting a plurality of feature quantities from a plurality of training data using a machine learning model; a prediction means for generating a prediction result regarding a task from the training data and teacher data corresponding to the training data; A negative example regularization means for calculating a similarity between the plurality of feature amounts; An information processing program for functioning as a training means for updating parameters of the machine learning model so that the features move away from each other based on the prediction result and the similarity.

Images