Machine learning program, machine learning device, and machine learning method

Abstract

The present invention addresses the problem that Attentions (regions of attention / interest) observed in an image by a plurality of Multi-Head Attention (MHA) heads in a ViT architecture may overlap and a head is likely to be wasted. A machine learning program according to the present invention causes a computer to execute a process for training a machine learning model having a plurality of mechanisms, each of which generates information of interest, such that the machine learning model calculates a cosine similarity of the information of interest generated by each of the plurality of mechanisms, calculates entropy of an aggregate value of the information of interest generated by the plurality of mechanisms, and minimizes the cosine similarity and maximizes the entropy. This makes it possible to suppress overlapping of the regions of interest of the plurality of MHA heads.

Description

Machine learning program, machine learning device, and machine learning method

[0001] The present invention relates to machine learning techniques.

[0002] There are techniques for recognizing objects in images using machine learning. For example, Vision Transformer (ViT) uses the Encoder part of the Transformer used in language models to treat images as characters, and is used for classification tasks and object detection tasks.

[0003] Emerging Properties in Self-Supervised Vision Transformers, ICCV2021, https: / / arxiv.org / abs / 2104.14294

[0004] However, there is a problem that the attention (attention / region of interest) information, which indicates the region of interest in an image, generated by each of the multiple heads of the Multi Head Attention (MHA) of the ViT architecture may overlap, resulting in wasted heads. Note that this problem can occur not only in the ViT architecture but also in any machine learning model that uses multiple mechanisms to generate information indicating the region of interest in an image.

[0005] In one aspect, an object of the present invention is to suppress overlapping of regions of interest determined by a plurality of mechanisms that generate information indicating regions of interest in an image.

[0006] In one aspect, a machine learning program causes a computer to perform a process of calculating the cosine similarity of the attention information generated by each of multiple mechanisms for a machine learning model having multiple mechanisms that each generate attention information, calculating the entropy of the aggregate value of the attention information generated by the multiple mechanisms, and training the machine learning model so as to minimize the cosine similarity and maximize the entropy.

[0007] In one aspect, overlap of the regions of interest of multiple heads of the MHA can be suppressed.

[0008] FIG. 1 is a diagram showing an example of ViT. FIG. 2 is a diagram showing an example of an attention map when ViT is used. FIG. 3 is a diagram showing an example of a comparison of attention between supervised learning, which is a conventional technique, and DINO. FIG. 4 is a diagram showing an example of a comparison of attention shown by multi-head between supervised learning, which is a conventional technique, and DINO. FIG. 5 is a diagram for explaining an example of a problem with DINO, which is a conventional technique. FIG. 6 is a diagram showing an example of the configuration of a machine learning device 10 according to this embodiment. FIG. 7 is a flowchart showing an example of the flow of a training process according to this embodiment. FIG. 8 is a diagram showing an example of accuracy by finetune for each method. FIG. 9 is a diagram showing an example of the effect of each method. FIG. 10 is a diagram showing an example of quantitative evaluation of each method. FIG. 11 is a diagram showing an example of the hardware configuration of the machine learning device 10 according to this embodiment.

[0009] Below, examples of the machine learning program, machine learning device, and machine learning method according to this embodiment will be described in detail with reference to the drawings. Note that this embodiment is not limited to these examples. Furthermore, each example can be combined as appropriate within a consistent range.

[0010] First, a machine learning model in this embodiment will be described. The machine learning model in this embodiment has multiple mechanisms that each generate information indicating a region of interest in an image when performing tasks such as image classification and object detection. As an example of such a machine learning model, this embodiment will explain using Vision Transformer (ViT), a deep learning model. FIG. 1 is a diagram illustrating an example of ViT. ViT processes images, for example, into sequence data (continuous data) similar to text, which is ViT's specialty. In the example of FIG. 1, ViT treats one image as sequence data consisting of nine 3x3 image patches. For example, ViT converts two-dimensional image data into one-dimensional sequence data for each patch, and performs processing such as obtaining correspondences between patches using a Transformer Encoder with the sequence data of each patch as input data.

[0011] Furthermore, the Transformer Encoder has a Multi Head Attention (MHA) section, which is an architecture that indicates where in an image multiple heads should focus their attention, as shown by the dashed lines in Figure 1. Here, each of the multiple heads in the MHA is a mechanism that generates information indicating an area of ​​attention in an image. An MHA typically has 6 to 12 heads, and there is a problem that the attention (attention / area of ​​attention) that each head focuses on overlaps, making the heads prone to waste. Here, attention in the MHA and Vision Transformer refers to information (attention information) that indicates an area of ​​attention in image data.

[0012] Figure 2 shows an example of an attention map when using ViT. Figure 2 visualizes an attention map of an image of a dog, cat, and airplane using ViT, which uses general supervised learning. Figure 2 shows the original image on the right and a visualized image of the attention map on the left. As shown in Figure 2, for example, attention is concentrated in areas that would not normally require attention, such as the edges of the image. This means that, for example, with ViT, which uses general supervised learning, it is not easy to understand the attention map through visualization, and that feature extraction is not able to be decomposed for each MHA on the attention side of feature extraction.

[0013] This section will explain attention using conventional techniques in more detail. Figure 3 is a diagram showing an example of a comparison of attention between conventional supervised learning and DINO. DINO shown in Figure 3 is a conventional technique shown in, for example, Non-Patent Document 1, which introduces self-supervised learning using distillation into ViT. In addition, in DINO, for example, as shown on the right side of Figure 3, different data extensions are applied to the teacher and student, and centering is performed to align the center of each feature to 0. As shown on the left side of Figure 3, DINO is able to focus attention on objects, although its accuracy is lower than that of supervised learning, for example.

[0014] Next, we will compare the attention of supervised learning for each multi-head with that of DINO. Figure 4 shows an example of a comparison of the attention shown by multi-heads in conventional supervised learning and DINO. The example in Figure 4 shows the attention of the final layer multi-head extracted from eight types of images, and shows three of the 12 heads of the ViT-base MHA. As shown in the example in Figure 4, for example, DINO can extract the attention shown by each multi-head more accurately than supervised learning.

[0015] However, the conventional technology DINO also has the following problems. Figure 5 is a diagram illustrating an example of the problems with the conventional technology DINO. Figure 5 shows a comparison example of the attention extracted by DINO for each multi-head from an image of a bird shown in the upper left, with the lower section showing the attention extracted by Head 1, Head 2, Head 4, and Head 5 out of multiple multi-heads. Referring to these, it can be seen that Head 4 has captured features including the surrounding environment, but Head 1, Head 2, and Head 5 show little difference, resulting in overlapping feature extraction.

[0016] Therefore, one of the objectives of this embodiment is to suppress overlap of attention regions of multiple heads of MHA. For example, in this embodiment, in the case of a person, attention is explicitly divided for each head, such as the head, legs, and torso, so that each head can pinpoint and focus on a different region of the image.

[0017] Therefore, in this embodiment, for example, the entropy of the attention of each head of the MHA is minimized, or the cosine similarity is minimized. This allows, for example, each head of the MHA to focus on a specific part, or constraints can be imposed so that the weights of each head are different, making it possible to suppress overlap of the attention of each head. Furthermore, in this embodiment, for example, the entropy of the aggregated value of the attention of all heads of the MHA is maximized. This makes it possible, for example, to train ViT so that information from the entire image is used evenly and attention weights are applied to different positions in each head of the MHA.

[0018] [Functional Configuration of Machine Learning Device 10] The functional configuration of the machine learning device 10, which is the executing entity of this embodiment, will be described using Figure 6. Figure 6 is a diagram showing an example configuration of the machine learning device 10 according to this embodiment. The machine learning device 10 shown in Figure 6 is, for example, an information processing device such as a desktop personal computer (PC), a notebook PC, or a server computer. Note that while Figure 6 shows the machine learning device 10 as a single computer, it may also be a distributed computing system made up of multiple computers. Furthermore, the machine learning device 10 may also be a cloud computing device managed by a service provider that provides cloud computing services.

[0019] As shown in FIG. 6 , the machine learning device 10 includes a communication unit 20 , a storage unit 30 , and a control unit 40 .

[0020] The communication unit 20 is a processing unit that controls communication with other devices, and is, for example, a communication interface such as a network interface card or a USB (Universal Serial Bus) interface.

[0021] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and stores, for example, model information 31 and the like.

[0022] The model information 31 stores, for example, information about ViT, which is widely used in image classification and object detection models, as well as model parameters and training data for constructing the ViT.

[0023] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0024] The control unit 40 is a processing unit, such as a processor, that controls the entire machine learning device 10. The control unit 40 includes a calculation unit 41 and a training unit 42. Each processing unit is an example of an electronic circuit included in a processor or an example of a process executed by a processor.

[0025] For example, the calculation unit 41 calculates the cosine similarity of the attention information generated by each of the mechanisms for a machine learning model having multiple mechanisms that each generate attention information. The calculation unit 41 also calculates the cosine similarity of the attention (attention / attention area) of each head of an MHA in a Transformer deep learning model, for example. The calculation unit 41 also calculates the entropy of the aggregate value of the attention information generated by the multiple mechanisms of the machine learning model. The calculation unit 41 also calculates the entropy of the aggregate value of the attention of all heads of the MHA. The calculation unit 41 also calculates a second entropy of the attention information generated by the multiple mechanisms of the machine learning model. The calculation unit 41 also calculates a second entropy of the attention of each head of the MHA, for example.

[0026] The training unit 42 trains a machine learning model such as a Transformer, for example, to minimize the cosine similarity of the attention of each head of the MHA and maximize the entropy of the aggregate value of the attention of all heads of the MHA, calculated by the calculation unit 41. Furthermore, the training unit 42 trains the machine learning model such as a Transformer, for example, to minimize a second entropy of the attention of each head of the MHA.

[0027] Furthermore, the process of minimizing the cosine similarity of the attention information generated by each of the multiple mechanisms of the machine learning model includes, for example, a process of minimizing the loss by taking the average value of the cosine similarity of the weights of the attention information generated by the multiple mechanisms as the loss. Furthermore, the process of minimizing the cosine similarity of the attention of each head of the MHA includes, for example, a process of minimizing the loss by taking the average value of the cosine similarity of the attention weights of each head of the MHA as the loss.

[0028] The minimization of the cosine similarity will be explained in more detail. For example, if two weight matrices A i and A j Assume that the cosine similarity of is defined as in the following equation (1).

[0029]

[0030] When formula (1) is defined in this way, for example, the loss L sim is the average of the cosine similarity of each attention for the N multi-heads (final block) of the Transformer, and can be expressed as the following equation (2).

[0031]

[0032] Therefore, the training unit 42 calculates the loss L using, for example, equation (2). sim Train the Transformer to minimize

[0033] Furthermore, the process of minimizing the second entropy of the attention information generated by the multiple mechanisms of the machine learning model includes, for example, a process of minimizing a loss by taking the sum of the Shannon entropy for the weight of the attention information generated by the multiple mechanisms as a loss. Furthermore, the process of minimizing the second entropy of the attention of each head of the MHA includes, for example, a process of minimizing the loss by taking the sum of the Shannon entropy for the attention weight of each head of the MHA as a loss.

[0034] The minimization of the second entropy will be described in more detail. For example, the training unit 42 calculates the attention weights A i For (i\in{1,2,...3,N}), we train a Transformer to minimize the total loss L of the Shannon entropy H, as shown in the following equation (3).

[0035]

[0036] In addition, in equation (3), H(A i ) is expressed, for example, by the following equation (4):

[0037]

[0038] However, in equation (4), A i,j,k is, for example, the output matrix element j,k of head i, and the base of log is 2 or e.

[0039] Furthermore, the process of maximizing the entropy of the aggregated value of attention information generated by multiple mechanisms of the machine learning model includes, for example, a process of maximizing a loss by taking the sum of Shannon entropy for the weights of the attention information generated by the multiple mechanisms as the loss. Furthermore, the process of maximizing the entropy of the aggregated value of attention of all heads of the MHA includes, for example, a process of maximizing a loss by taking the sum of Shannon entropy for the attention weights of all heads as the loss.

[0040] The maximization of the entropy will be explained in more detail. For example, the Shannon entropy is calculated by the weight matrix A iIf the input sequence length of the Transformer for (i^in{1,2,...,N}) is L and each sequence is l, then it can be expressed by the following equation (5). Note that the sequence here means, for example, patches obtained by dividing each image into rectangles.

[0041]

[0042] The training unit 42 calculates, for example, the loss H se The Transformer is trained to maximize B. In equation (5), B is expressed by, for example, the following equation (6).

[0043]

[0044] [Processing Flow] Next, the flow of training processing by the machine learning device 10 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the flow of training processing according to this embodiment.

[0045] First, the machine learning device 10 creates, for example, a Vit model (step S101).

[0046] Next, the machine learning device 10 forwards the image data to the Vit model created in step S101 (step S102), for example.

[0047] Next, when the number of epochs reaches a predetermined number (step S103: Yes), the training process shown in FIG. 7 ends.

[0048] On the other hand, if the number of epochs has not reached a predetermined number (step S103: No), the machine learning device 10 calculates a loss function (A) from the output of ViT that forwarded the image data in step S102 (step S104). Here, the loss function (A) is, for example, the cross-entropy loss with the correct label in the case of supervised learning, or the KL loss or cross-entropy loss between the teacher model and the student model in the case of self-supervised learning, which is pre-learning using distillation.

[0049] Next, the machine learning device 10 calculates the MHA diversification loss (B) from the MHA weights of the current ViT (step S105). Here, the MHA diversification loss (B) is, for example, the loss shown in the above-mentioned formula (2) or formula (3).

[0050] Next, the machine learning device 10 modifies the weights of ViT by backpropagation using the gradient descent method so as to minimize the total loss of the loss function (A) + the MHA diversification loss (B) (step S106). Note that, for example, at this time, the weights of ViT may be further modified so as to maximize the loss shown in the above-mentioned formula (5).

[0051] Next, the machine learning device 10, for example, increments the epoch (step S107) and repeats the process until the number of epochs reaches a predetermined number (steps S103 to S107).

[0052] [Effects] Next, for reference, the effects of the training process according to this embodiment will be described. Note that, between a training method according to this embodiment that minimizes the cosine similarity of the attention of each head of the MHA and a training method that minimizes the entropy of the attention of each head of the MHA, the former was found to have a superior effect on accuracy, etc. Therefore, below, the effects of the training method according to this embodiment that minimizes the entropy of the attention of each head of the MHA will be omitted, and the effects of the training method that minimizes the entropy of the attention of each head of the MHA will be described.

[0053] FIG. 8 shows an example of the accuracy achieved by finetune for each method. In FIG. 8, DINO (vanilla) is a training method using DINO, a conventional technology described in, for example, Non-Patent Document 1. Also, in FIG. 8, Cos Similarity↑ is a training method that minimizes the cosine similarity of attention for each head of the MHA according to this embodiment. Also, Cos Similarity↑ + Entropy↓ and Cos Similarity↑ + Entropy↓↓ are training methods that minimize the cosine similarity of attention for each head of the MHA according to this embodiment and maximize the entropy of the aggregated value of attention for all heads. The arrows ↑ and ↓ indicate the strength of regularization (hyperparameters). Also, in FIG. 8, for example, Linear is a training method evaluated by validation after finetuning using 100 epochs, one fully connected layer, and training, and KNN is a training method evaluated by validation after training. Referring to FIG. 8, it can be seen that the accuracy achieved by each method is roughly the same, and that Cos similarity↑+Entropy↓ has the best accuracy.

[0054] Fig. 9 is a diagram showing an example of the effect of each technique. Fig. 9 is a diagram showing an example of a comparison of attention shown by multi-heads (each for 6 heads) of each technique in Vit-S patch size 16 head 6 300 epochs. Referring to Fig. 9, compared to the conventional technology DINO (vanilla), the three techniques according to this embodiment are able to suppress overlap of attention by each head, although visually only slightly (for example, not focusing only on the bird's beak).

[0055] Fig. 10 is a diagram showing an example of quantitative evaluation of each method. Fig. 10 shows the similarity (cos similarity) between each head of the MHA when each method is used as a quantitative evaluation of each method.

[0056] In FIG. 10 , DINO-Vanilla is a training method using DINO, a conventional technology disclosed in, for example, Non-Patent Document 1. Also in FIG. 10 , DINO-Sim is a training method that minimizes the cosine similarity of attention for each head of the MHA according to this embodiment. Also in FIG. 10 , DINO-Ent is a training method that maximizes the entropy of the aggregated value of attention for all heads of the MHA according to this embodiment. Also in FIG. 10 , DINO-Sim-Ent is a training method that minimizes the cosine similarity of attention for each head of the MHA according to this embodiment and maximizes the entropy of the aggregated value of attention for all heads.

[0057] 10, it can be seen that training to minimize the cosine similarity of attention for each head of the MHA results in a final model weight with small cosine similarity between heads. In particular, it can be seen that DINO-Sim-Ent has the smallest cosine similarity, and is trained to have greater diversity.

[0058] Furthermore, as described above, for a machine learning model having multiple mechanisms that each generate information of interest, the machine learning device 10 calculates the cosine similarity of the information of interest generated by each of the multiple mechanisms, calculates the entropy of the aggregate value of the information of interest generated by the multiple mechanisms, and trains the machine learning model to minimize the cosine similarity and maximize the entropy.

[0059] In this way, the machine learning device 10 trains the machine learning model to minimize the cosine similarity of the attention information generated by each of the multiple mechanisms and maximize the entropy of the aggregate value of the attention information generated by the multiple mechanisms. This allows the machine learning device 10 to suppress overlap of the attention regions of the multiple heads of the MHA.

[0060] In addition, the machine learning device 10 causes a computer to execute a process of calculating a second entropy of attention information generated by multiple mechanisms, and the process of training a machine learning model includes a process of training the machine learning model so as to minimize the cosine similarity and the second entropy and maximize the entropy.

[0061] This allows the machine learning device 10 to further reduce overlap of the regions of interest of multiple heads of the MHA.

[0062] Furthermore, the process of minimizing the cosine similarity executed by the machine learning device 10 includes a process of minimizing a loss using the average value of the cosine similarities of the weights of the attention information generated by multiple mechanisms as the loss.

[0063] This allows the machine learning device 10 to suppress overlapping of the regions of interest of multiple heads of the MHA.

[0064] Furthermore, the process of minimizing the second entropy executed by the machine learning device 10 includes a process of minimizing a loss using the total value of the Shannon entropy for the weights of the attention information generated by multiple mechanisms as the loss.

[0065] This allows the machine learning device 10 to suppress overlapping of the regions of interest of multiple heads of the MHA.

[0066] Furthermore, the process of maximizing entropy executed by the machine learning device 10 includes a process of maximizing a loss using the total value of Shannon entropy for the weight of the attention information generated by a plurality of mechanisms as the loss.

[0067] This allows the machine learning device 10 to suppress overlapping of the regions of interest of multiple heads of the MHA.

[0068] [System] The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, may be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and may be changed as desired.

[0069] Furthermore, the specific form of distribution or integration of the components of the machine learning device 10 is not limited to that shown in the figure. For example, the training unit 42 of the machine learning device 10 may be distributed across multiple processing units, or the calculation unit 41 and training unit 42 of the machine learning device 10 may be integrated into a single processing unit. That is, all or some of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0070] 11 is a diagram illustrating an example of the hardware configuration of a machine learning device 10 according to this embodiment. As shown in FIG. 11, the machine learning device 10 includes a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components illustrated in FIG. 11 are connected to each other via a bus or the like.

[0071] The communication interface 10a is a network interface card or the like, and communicates with other information processing devices. The HDD 10b stores, for example, programs and data for operating the various functions shown in FIG.

[0072] The processor 10d is a CPU, a microprocessing unit (MPU), a graphics processing unit (GPU), or the like. The processor 10d may also be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 10d reads, for example, from the HDD 10b, a program that executes the same processes as those of the processing units shown in FIG. 6, and loads the program into the memory 10c. This allows the processor 10d to operate as a hardware circuit that executes processes that implement the functions described in FIG. 6, etc.

[0073] Furthermore, the machine learning device 10 can also realize functions similar to those of the above-described embodiments by reading the program from a recording medium using a media reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the machine learning device 10. For example, the above-described embodiments may also be applied to cases where an information processing device other than the machine learning device 10 executes the program, or where the machine learning device 10 and another information processing device cooperate to execute the program.

[0074] The program may be distributed via a network such as the Internet. The program may also be recorded on a computer-readable storage medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD). The program may then be read from the recording medium and executed by the machine learning device 10 or the like.

[0075] REFERENCE SIGNS LIST 10 machine learning device 10a communication interface 10b HDD 10c memory 10d processor 20 communication unit 30 storage unit 31 model information 40 control unit 41 calculation unit 42 training unit

Claims

1. A machine learning program that causes a computer to execute the following processes: for a machine learning model having multiple mechanisms that each generate information of interest, calculate a cosine similarity of the information of interest generated by each of the multiple mechanisms; calculate an entropy of an aggregate value of the information of interest generated by the multiple mechanisms; and train the machine learning model so as to minimize the cosine similarity and maximize the entropy.

2. The machine learning program of claim 1, further comprising: causing the computer to execute a process of calculating a second entropy of the attention information generated by the multiple mechanisms; and training the machine learning model, the process including a process of training the machine learning model to minimize the cosine similarity and the second entropy, and maximize the entropy.

3. The machine learning program according to claim 1 or 2, characterized in that the process of minimizing the cosine similarity includes a process of minimizing a loss by setting the average value of the cosine similarity of the weights of the attention information generated by the multiple mechanisms as a loss.

4. The machine learning program according to claim 2, characterized in that the process of minimizing the second entropy includes a process of minimizing a loss, the loss being the sum of the Shannon entropies for the weights of the attention information generated by the multiple mechanisms.

5. The machine learning program according to claim 1 or 2, characterized in that the process of maximizing the entropy includes a process of maximizing a loss by setting the sum of the Shannon entropy for the weight of the attention information generated by the multiple mechanisms as a loss.

6. A machine learning device comprising a control unit that executes the following processes: for a machine learning model having a plurality of mechanisms each generating information of interest, calculates a cosine similarity of the information of interest generated by each of the plurality of mechanisms; calculates an entropy of an aggregate value of the information of interest generated by the plurality of mechanisms; and trains the machine learning model so as to minimize the cosine similarity and maximize the entropy.

7. A machine learning method in which a computer executes the following processes: for a machine learning model having a plurality of mechanisms each generating information of interest, calculate a cosine similarity of the information of interest generated by each of the plurality of mechanisms; calculate an entropy of an aggregate value of the information of interest generated by the plurality of mechanisms; and train the machine learning model so as to minimize the cosine similarity and maximize the entropy.

Images