Information processing apparatus, method, and program

Abstract

To improve the performance while reducing the model size.SOLUTION: An information processing apparatus includes an acquisition unit, a determination unit, a first extraction unit, a second extraction unit, and a training unit. The acquisition unit acquires training data to be used for training a first feature extractor and a second feature extractor. The determination unit determines a model size of the second feature extractor. The first extraction unit inputs the training data to the first feature extractor to extract first feature quantity. The second extraction unit inputs the first feature quantity to the second feature extractor to extract second feature quantity. The training unit trains the first feature extractor so that the first feature quantity may be close to the second feature quantity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] In deep learning, a power law has been observed in which performance improves with model size in natural language processing and image recognition processing. However, there is a problem that the larger the model size, the higher the memory and computational costs during inference. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., Ma, K., Be your own teacher: Improve the performance of convolutional neural networks via self distillation., Proceedings of the IEEE / CVF International Conference on Computer Vision. (2019) 3713-3722 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been made to solve the above-mentioned problems, and has an object to provide an information processing device, method, and program that can improve performance while reducing model size. [Means for solving the problem]

[0005] The information processing device according to this embodiment includes an acquisition unit, a determination unit, a first extraction unit, a second extraction unit, and a training unit. The acquisition unit acquires training data used to train a first feature extractor and a second feature extractor. The determination unit determines a model size of the second feature extractor. The first extraction unit inputs the training data to the first feature extractor and extracts a first feature. The second extraction unit inputs the first feature to the second feature extractor and extracts a second feature. The training unit trains the first feature extractor so as to bring the first feature closer to the second feature. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an information processing apparatus according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of the operation of the information processing device according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing a first example of a first feature extractor and a second feature extractor. [Figure 4] FIG. 13 is a diagram showing a second example of the first feature extractor and the second feature extractor. [Diagram 5] FIG. 11 is a block diagram showing an information processing apparatus according to a second embodiment. [Figure 6] 10 is a flowchart showing an example of the operation of the information processing device according to the second embodiment. [Figure 7] FIG. 11 is a block diagram showing an information processing apparatus according to a third embodiment. [Figure 8] 13 is a flowchart showing a first operation example of the information processing device according to the third embodiment. [Figure 9] 13 is a flowchart showing a second operation example of the information processing device according to the third embodiment. [Figure 10] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an information processing device, method, and program according to the present embodiment will be described in detail with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and duplicated descriptions will be omitted as appropriate.

[0008] (First embodiment) The information processing apparatus according to the first embodiment will be described with reference to the block diagram of FIG. The information processing device 10 according to the first embodiment includes a storage unit 101, an acquisition unit 102, a determination unit 103, a first extraction unit 104, a second extraction unit 105, a training unit 106, and an output unit 107.

[0009] The storage unit 101 stores a machine learning model (hereinafter, simply referred to as a model) having a first feature extractor and a second feature extractor, training data, and the like. The model is assumed to be, for example, a Vision Transformer (ViT), a Bidirectional Encoder Representations from Transformer (BERT), a Generative Pretrained Transformer (GPT), and the like, which use a so-called Transformer structure. Note that the model is not limited to the Transformer structure, and may be a Multi Layer Perceptron (MLP)-Mixer, and any structure may be used as long as the number of dimensions of the input to the model and the output from the model are the same. This structure makes it easy to stack layers that form the model.

[0010] In this embodiment, the training data is assumed to be images, but is not limited to this. For example, the training data may be one-dimensional time series data such as control signal data, voice data, natural language data, music data, and sensor values, or two-dimensional or higher data other than images such as symbol data, log data, video data, table data, and transaction data.

[0011] The acquiring unit 102 acquires training data used for training the first feature extractor and the second feature extractor. The determination unit 103 determines the model size of the second feature extractor. The first extraction unit 104 inputs the training data to a first feature extractor and extracts a first feature. The second extraction unit 105 inputs the first feature amount to a second feature amount extractor and extracts a second feature amount. The training unit 106 trains the first feature extractor so as to bring the first feature closer to the second feature. Specifically, the training unit 106 updates parameters of the first feature extractor. After the training is completed, a trained first feature extractor is generated. The parameters include, for example, network weights and biases. The output unit 107 outputs the trained first feature extractor to, for example, an external device.

[0012] Next, an example of the operation of the information processing device 10 according to the first embodiment will be described with reference to the flowchart of FIG.

[0013] In step SA 1 , the acquisition unit 102 acquires training data from the storage unit 101 . In step SA2, the determination unit 103 determines the model size of the second feature extractor. The model size may be determined based on at least one of the size that can be mounted on the memory size of a semiconductor chip (CPU / GPU / TPU, etc.), the calculation cost, and the inference accuracy. Specifically, when the memory size of the semiconductor chip of the computer is large and a high calculation cost is acceptable, the model size of the second feature extractor may be set large. In addition, since the larger the model size, the higher the inference accuracy is generally, the model size of the second feature extractor may be determined so as to achieve the required inference accuracy.

[0014] In step SA3, the first extraction unit 104 extracts a first feature using a first feature extractor. Specifically, the training data is input to the first feature extractor, and the first feature is extracted as an output of the first feature extractor. In step SA4, the second extraction unit 105 extracts a second feature using a second feature extractor. Specifically, the first feature is input to the second feature extractor, and a third feature is extracted as an output of the second feature extractor.

[0015] In step SA5, the training unit 106 calculates a loss using a loss function for bringing the first feature closer to the second feature. For example, the first feature for the i-th (i>0) training data is calculated as v i Let the second feature be z i Then, the loss L representing the distance between the first feature and the second feature is d can be expressed as (1), where N is the number of training data or mini-batches.

[0016]

number

[0017] In step SA6, it is determined whether the training of the first feature extractor is completed. The determination of whether the training is completed is performed, for example, by using the loss function L d If the loss L is less than or equal to a threshold, training is determined to be complete. d Training may be determined to be complete when the decrease in converges. Furthermore, training may be determined to be complete when a predetermined number of epochs have been processed. If training is complete, proceed to step SA8, and if training is not complete, proceed to step SA7.

[0018] In step SA7, the training unit 106 calculates the loss L d The parameters of the model (here, the first feature extractor) are updated, for example, by gradient descent and backpropagation, so as to minimize the loss L; specifically, parameters such as the weights and biases of the network related to the first feature extractor are updated. After the parameters are updated, the process returns to step SA1, and training is continued using new training data. When training is finally completed, the training of the first feature extractor ends, and a trained first feature extractor (also simply referred to as a trained model) is generated. The loss Ld By training the first feature extractor so that θ is minimized, the first feature can be made to approach the second feature. In step SA8, the output unit 107 outputs the trained first feature extractor. The output unit 107 may output the trained first feature extractor to an external device (not shown) such as a server, or may store the trained first feature extractor in the storage unit 101 of the information processing device 10.

[0019] In the example flowchart of FIG. 2, the first feature and the second feature are compared, but an MLP head may be connected to each of the first feature extractor and the second feature extractor, and the first feature may be brought closer to the second feature based on a predicted value obtained from the MLP head. Specifically, for example, the first feature value v i is input to the MLP head, and the output from the MLP head is v' i In this case, the loss L1 of the first feature extractor can be expressed by equation (2).

[0020]

number

[0021] where CE is the cross entropy, σ is the softmax function, and y is the class label of the training data. Similarly, the second feature z i is input to the MLP head, and the output from the MLP head is z' i In this case, the loss L2 of the second feature extractor for the class label y of the training data can be expressed by equation (3).

[0022]

number

[0023] The parameters (weights, biases, etc.) of the MLP head connected to the first feature extractor and the MLP head connected to the second feature extractor may be the same, or different parameters may be set. Based on equations (1) to (3), the total loss L can be determined as shown in equation (4).

[0024]

number

[0025] where α, β, and γ are coefficients that determine the weight of each loss. The training unit 106 trains the model to reduce the overall loss L defined by equation (4), thereby training the first feature extractor to bring the first feature closer to the second feature.

[0026] Next, a first example of the first feature extractor and the second feature extractor used in the information processing device 10 according to the first embodiment will be described with reference to FIG. 3 includes a first feature extractor 31 and a second feature extractor 32. It is assumed that the first feature extractor 31 and the second feature extractor 32 are configured by ViT.

[0027] The first feature extractor 31 includes a patch embedding process (Patch Embedding) 31-1, an add position embedding process (Add Position Embedding) 31-2, and one or more Transformer Encoder blocks 31-3.

[0028] Patch images 34 obtained by dividing an image 33 are embedded into a specified number of dimensions by a patch embedding process 31-1. Next, position embedding process 31-2 embeds position information indicating the position of the patch image 34 in the entire image (input image 33) and a learnable class token 35 into the patch image 34. As the class token 35 is assumed to be a token commonly used in ViT, a detailed description thereof will be omitted.

[0029] Then, the patch image with the embedded position information is processed by one or more Transformer Encoder blocks 31-3 to extract a first feature 36-1. The first feature 36-1 can be said to be a feature from an intermediate layer. Note that the Transformer Encoder block 31-3 itself is a block used in general ViT, so a detailed description of its operation will be omitted.

[0030] The second feature extractor 32 includes one or more Transformer Encoder blocks 31-3. The second feature extractor 32 has the same configuration as the Transformer Encoder block 31-3 of the first feature extractor 31, and therefore can be easily stacked. 3 shows an example in which the first feature extractor 31 and the second feature extractor 32 include the same number of Transformer Encoder blocks 31-3, that is, two layers, but the present invention is not limited to this and each may include three or more layers. Furthermore, the number of layers of the Transformer Encoder blocks 31-3 of the first feature extractor 31 and the second feature extractor 32 may be different. Furthermore, the number of layers of the Transformer Encoder blocks 31-3 of either the first feature extractor 31 or the second feature extractor 32 may be greater.

[0031] The Transformer Encoder block 31-3 of the second feature extractor 32 receives the first feature 36-1 output from the first feature extractor 31, and outputs a second feature 36-2. The training unit 106 may use the above-mentioned formula (1) to train the first feature extractor 31 so as to bring the first feature 36-1 closer to the second feature 36-2. As a result, during training, training is performed with the second feature extractor 32 connected, and during inference, highly accurate inference can be performed using only the first feature extractor 31 without using the second feature extractor 32.

[0032] Next, a second example of the first feature extractor and the second feature extractor used in the information processing device 10 according to the first embodiment will be described with reference to FIG. 4 differs from FIG. 3 in that an MLP head 41-1 is connected to the first feature extractor 31, and an MLP head 41-2 is connected to the second feature extractor 32. In FIG. The MLP head 41-1 receives a first feature and outputs a predicted value according to the task. In this case, assuming a classification task, the MLP head 41-1 outputs an image of "Fox" for the input image "Cat" as the predicted value 42-1. On the other hand, the MLP head 41-2 receives a second feature and outputs a predicted value according to the task. In this case, the MLP head 41-2 outputs an image of "Cat" for the input image "Cat" as the predicted value 42-2.

[0033] The training unit 106 uses the loss functions from the above-mentioned formulas (2) to (4) to train the first feature extractor 31 while the second feature extractor 32 is connected so as to bring the predicted value 42-1 closer to the predicted value 42-2. In other words, training should be performed so as to bring the outputs of the class tokens related to the second feature extractor 32 and the class tokens related to the first feature extractor 31 closer to each other.

[0034] As the class label y in the formulas (2) and (3), the input image 33 may be used. Although an example of a classification task is shown here, the present invention is not limited to this, and other tasks such as an object detection task, an image generation task, and anomaly detection task can be executed in the same manner.

[0035] According to the first embodiment described above, training data is input to a first feature extractor to extract first features, the first features are input to a second feature extractor to extract second features, and the first feature extractor is trained so as to bring the first features closer to the second features. That is, training is performed using a large model that connects the first feature extractor and the second feature extractor, and during inference, the second feature extractor is removed and a small model with only the first feature extractor is used. This makes it possible to generate a model with inference accuracy equivalent to that of the large model using only the first feature extractor. In other words, performance can be improved while reducing the model size.

[0036] Second embodiment The information processing device 10 according to the second embodiment differs from the first embodiment in that pre-trained weights are used.

[0037] A block diagram of an information processing device 10 according to the second embodiment will be described with reference to FIG. The information processing device 10 shown in Fig. 5 is similar to the information processing device 10 shown in Fig. 1 except that a pre-trained parameter 51 is stored in a storage unit 101, and therefore a description thereof will be omitted. The pre-trained parameters are, for example, parameters such as weights and biases that are learned in advance by the first feature extractor based on training data. When the second feature extractor is initialized by a predetermined method, a part or the whole of the second feature extractor uses pre-trained parameters.

[0038] Next, an example of the operation of the information processing device 10 according to the second embodiment will be described with reference to the flowchart of FIG. Steps SA1 to SA8 are the same as those in FIG. 2, so the description will be omitted.

[0039] In step SB1, the determination unit 103 or the training unit 106 acquires pre-trained parameters from the storage unit 101 and sets the pre-trained parameters as parameters of the first feature extractor. Specifically, the training unit 106 determines weights and biases from the pre-trained parameters.

[0040] According to the second embodiment described above, the pre-trained parameters are set as the parameters of the first feature extractor, so that the parameters having a certain degree of inference accuracy in advance become the initial values, thereby shortening the training time of the first feature extractor and enabling further improvement in inference accuracy.

[0041] Third embodiment The third embodiment differs from the above-described embodiments in that a plurality of second feature extractors different from each other are prepared.

[0042] FIG. 7 shows a block diagram of an information processing device 10 according to the third embodiment. The information processing device 10 shown in FIG. 7 is similar to the information processing device 10 shown in FIG. 1 except for the processing of the storage unit 101 and the determination unit 103, and therefore a description thereof will be omitted.

[0043] The storage unit 101 stores a plurality of second feature extractors 71. The stored second feature extractors 71 are different from each other in at least one of the model size and the parameters such as the weight and the bias. The determining unit 103 selects a second feature extractor to be connected to the first feature extractor from among the second feature extractors stored in the storage unit 101 .

[0044] Next, a first operation example of the information processing device 10 according to the third embodiment will be described with reference to the flowchart of FIG. Steps SA1, SA3 to SA8 are the same as those in FIG. 2, and therefore their explanation will be omitted.

[0045] In step SC1, the determination unit 103 selects a second feature extractor, and connects the first feature extractor and the selected second feature extractor to design a model for training. As a result, the determination unit 103 only needs to select the second feature extractor from the storage unit 101, and therefore does not need to calculate and determine the model size of the second feature extractor, etc., each time during the model training process.

[0046] Alternatively, the user may select the second feature extractor. For example, the computational resources and costs are associated with the second feature extractor having a model size corresponding to the computational resources and costs. For example, information on the association may be displayed on a GUI, and the user may select a second feature extractor according to the desired computational resources and costs, so that the determination unit 103 may select the second feature extractor in response to an instruction from the user.

[0047] Note that a first feature extractor may be trained using multiple second feature extractors to generate multiple first feature extractors for the ensemble. A second operation example of the information processing device 10 according to the third embodiment will be described with reference to the flowchart of FIG. Step SA1, steps SA3 to SA7, and step SC1 are similar to those in FIG. 8, and therefore their explanation will be omitted.

[0048] In step SC2, the training unit 106 judges whether the second feature extractor has been switched a predetermined number of times. That is, it judges whether a predetermined number of second feature extractors have been selected and trained from the storage unit 101. If the second feature extractors have been switched a predetermined number of times, the process proceeds to step SC4, and if the second feature extractors have not been switched a predetermined number of times, the process proceeds to step SC3.

[0049] In step SC3, the determination unit 103 selects a new second feature extractor, connects the first feature extractor and the new second feature extractor, and designs a model for the next training. After that, the processes from step SA3 to step SA7 are repeated to train the model.

[0050] In step SC4, the output unit 107 outputs a plurality of trained first feature extractors. For example, during inference, the inference accuracy can be improved by ensemble processing using a plurality of trained first feature extractors. In the third embodiment, it is assumed that a plurality of second feature extractors are stored in the storage unit 101. However, this is not limiting, and a configuration may be adopted in which a plurality of second feature extractors are stored in an external device such as a server, and the determination unit 103 selects and acquires a second feature extractor from the external device.

[0051] In addition, if the model has the same number of dimensions of features in each layer, such as ViT, the number of layers can be easily increased or decreased, so that it can be appropriately applied to, for example, federated learning. That is, assume that the information processing device 10 receives parameters transmitted from each client and integrates the models for training. At this time, the determination unit 103 can appropriately deploy the first feature extractor with the model size increased or decreased according to the resource situation of the client.

[0052] According to the third embodiment described above, a plurality of second feature extractors are stored in the storage unit, and the determination unit selects a second feature extractor to be used for training from the plurality of second feature extractors. This not only reduces the model size while improving the model performance, but also eliminates the need to calculate the model size of the second feature extractor and determine it each time training is performed. In addition, a first feature extractor may be trained using multiple second feature extractors to generate an ensemble of first feature extractors, which can further improve inference accuracy.

[0053] Next, an example of a hardware configuration of the information processing device 10 according to each of the above-mentioned embodiments is shown in a block diagram of FIG. The information processing device 10 includes a CPU (Central Processing Unit) 91, a RAM (Random Access Memory) 92, a ROM (Read Only Memory) 93, a storage 94, a display device 95, an input device 96, and a communication device 97, each of which is connected by a bus.

[0054] The CPU 91 is a processor that executes arithmetic processing and control processing according to a program. The CPU 91 uses a predetermined area of ​​the RAM 92 as a working area and executes the processing of each part of the information processing device 10 described above in cooperation with programs stored in the ROM 93 and the storage 94. Note that each process of the information processing device 10 may be executed by one processor or may be executed in a distributed manner by multiple processors.

[0055] The RAM 92 is a memory such as a Synchronous Dynamic Random Access Memory (SDRAM), and functions as a work area for the CPU 91. The ROM 93 is a memory that stores programs and various information in a non-rewritable manner.

[0056] The storage 94 is a device that writes and reads data to a magnetic recording medium such as a hard disk drive (HDD), a semiconductor storage medium such as a flash memory, a magnetically recordable storage medium such as a HDD, an optically recordable storage medium, etc. The storage 94 writes and reads data to the storage medium in response to control from the CPU 91.

[0057] The display device 95 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 95 displays various information based on a display signal from the CPU 91. The input device 96 is an input device such as a mouse, a keyboard, etc. The input device 96 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 91. The communication device 97 communicates with external devices via a network under the control of the CPU 91 .

[0058] The instructions shown in the processing procedure shown in the above-mentioned embodiment can be executed based on a program, which is software. A general-purpose computer system can store this program in advance and obtain the same effect as the effect of the control operation of the above-mentioned information processing device by reading this program. The instructions described in the above-mentioned embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray (registered trademark) Disc, etc.), a semiconductor memory, or a recording medium similar thereto. The recording medium may be in any storage format as long as it is readable by a computer or an embedded system. The computer can realize the same operation as the control of the information processing device of the above-mentioned embodiment by reading the program from this recording medium and executing the instructions described in the program based on this program with a CPU. Of course, when the computer acquires or reads the program, it may acquire or read it through a network.

[0059] In addition, an OS (operating system), database management software, network, or other MW (middleware) running on a computer may execute some of the processes required to realize this embodiment based on instructions from a program installed on the computer or embedded system from a recording medium. Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like. Furthermore, the number of recording media is not limited to one, and a case where the processing in this embodiment is executed from a plurality of media is also included in the recording media in this embodiment, and the media may have any configuration.

[0060] The computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored in a recording medium, and may be configured as any one of a device such as a personal computer or a microcomputer, or a system in which multiple devices are connected to a network. In addition, the computer in this embodiment is not limited to a personal computer but also includes an arithmetic processing device, a microcomputer, etc. included in information processing equipment, and is a general term for equipment or devices that can realize the functions in this embodiment by a program.

[0061] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0062] 10...information processing device, 31...first feature extractor, 32,71...second feature extractor, 31-1...patch embedding process, 31-2...position embedding process, 31-3...Transformer Encoder block, 33...input image, 34...patch image, 35...class token, 36-1...first feature, 36-2...second feature, 41-1,41-2...MLP head, 42-1,42-2...prediction value, 51...pre-trained parameters, 91...CPU, 92...RAM, 93...ROM, 94...storage, 95...display device, 96...input device, 97...communication device, 101...storage unit, 102...acquisition unit, 103...determination unit, 104...first extraction unit, 105...second extraction unit, 106...training unit, 107...output unit.

Claims

1. an acquisition unit that acquires training data used for training the first feature extractor and the second feature extractor; A determination unit that determines a model size of the second feature extractor; a first extraction unit that inputs the training data to the first feature extractor and extracts a first feature; a second extraction unit that inputs the first feature to the second feature extractor and extracts a second feature; a training unit that trains the first feature extractor so as to approximate the first feature to the second feature; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the first feature extractor and the second feature extractor have a structure in which the number of dimensions of the extracted features is the same.

3. The information processing device according to claim 1 , wherein the first feature extractor is set to have pre-trained parameters as initial values.

4. A storage unit that stores a plurality of the second feature extractors, The information processing apparatus according to claim 1 , wherein the determination unit selects the second feature extractor to be used for training from among a plurality of second feature extractors.

5. The information processing device according to claim 1 , wherein the determination unit determines the model size of the second feature extractor based on at least one of a memory size, a calculation cost, and an inference accuracy.

6. The information processing apparatus according to claim 1 , wherein the first feature extractor and the second feature extractor are a model using a Transformer structure or a model using an MLP-Mixer.

7. The first extraction unit extracts, as the first feature, a feature of an intermediate layer of the first feature extractor; The information processing apparatus according to claim 1 , wherein the second extraction unit extracts, as the second feature, a feature of an intermediate layer of the second feature extractor.

8. obtaining training data used to train the first feature extractor and the second feature extractor; determining a model size of the second feature extractor; The training data is input to the first feature extractor to extract a first feature; extracting a second feature by inputting the first feature to the second feature extractor; training the first feature extractor so as to approximate the first feature to the second feature.

9. Computer, an acquisition means for acquiring training data used for training the first feature extractor and the second feature extractor; A determination means for determining a model size of the second feature extractor; a first extraction means for inputting the training data to the first feature extractor and extracting a first feature; a second extraction means for inputting the first feature to the second feature extractor and extracting a second feature; an information processing program for causing the first feature extractor to function as a training means for training the first feature extractor so as to bring the first feature closer to the second feature;

Images