A method and application of improving image recognition accuracy using orthogonal visual transformer
By limiting the projection matrix in the visual converter to an orthogonal matrix and using the full mapping of orthogonal groups, the optimization problem on manifolds into an Euclidean space is solved, and the accuracy of image recognition and training stability are improved.
Patent Information
- Application Number
- CN202210628383.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-24
- Filing Date
- 2022-06-06
- Publication Date
- 2025-02-28
- Estimated Expiration
- 2042-06-06
AI Technical Summary
Vision converters (ViT) have gradient vanishing problems in image recognition tasks, resulting in instability in training and slow optimization.
The projection matrix of the query matrix, key matrix and value matrix in the self-attention mechanism in the visual converter is limited to the orthogonal matrix, and the full mapping between the orthogonal group and its Li algebra is used to transform the optimization problem on manifold into the optimization problem on Euclidean space.
Through orthogonal constraints, the gradient explosion and gradient disappearance problems are alleviated, making training more stable, improving the accuracy of image recognition, and reducing training costs.
Smart Images
Figure CN114943883B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to Chinese patent application No. 202210566128.9 filed on May 24, 2022, the contents of which are incorporated herein by reference. Technical Field
[0003] The present invention belongs to the technical field of machine learning and visual converter algorithms, and relates to a method for improving image recognition accuracy by using an orthogonal visual converter for an image recognition scene, and an application thereof. Background Art
[0004] In recent years, various types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs), have been widely used in artificial intelligence fields such as computer vision and natural language processing. Transformer is also a type of neural network. Its self-attention mechanism can learn features in language sequences and has been successful in natural language processing. Inspired by Transformer, Vision Transformer (ViT) uses the self-attention mechanism to learn features of image patch sequences. Convolutional neural networks use convolution operations to expand the view of the convolution kernel layer by layer, that is, the deeper the network, the larger the features captured. Unlike convolutional neural networks, the self-attention mechanism enables ViT to obtain global features even in shallow networks. Therefore, in recent years, ViT has gradually replaced convolutional neural networks and has achieved great success in computer vision tasks such as image classification. However, ViT's self-attention mechanism has the risk of causing the gradient vanishing problem, which limits ViT from finding the optimal solution or slows down its optimization speed.
[0005] A manifold is a collection of objects subject to certain constraints, and different manifolds have different properties. In order to obtain certain properties, the parameters of the neural network can be restricted to a specific manifold. An orthogonal manifold is a commonly used specific manifold. Each matrix on an orthogonal manifold has the following property: the product of the matrix itself and its transposed matrix is the identity matrix. On the one hand, orthogonal transformations are numerically stable and will not widen the gap between data. On the other hand, orthogonal transformations do not compress or stretch the original space, thereby protecting the internal information of the original space from loss. Not only that, parameter orthogonalization also has the advantages of fast convergence and robustness. Therefore, restricting the projection matrix of the query matrix, key matrix, and value matrix in the traditional self-attention mechanism to an orthogonal manifold can maintain the feature space and alleviate the gradient vanishing problem. However, it also brings problems of high training cost and difficult gradient backpropagation: gradient backpropagation on the manifold requires updating the trainable parameters along the geodesic, involving orthogonal projection and retraction operations with large computational load.
[0006] Replacing optimization on the manifold with hard orthogonality constraints can reduce training costs, a strategy that has been used in orthogonal convolutional neural networks and recurrent neural networks. Hard orthogonality constraints use the difference between the product of the parameter matrix and its transpose matrix and the identity matrix as a regularization term for the main task. However, the search space of the main task does not necessarily intersect with the space of hard orthogonality constraints. Therefore, hard orthogonality constraints may not converge to the optimal point that satisfies both the main task and the orthogonality constraints. Another way to reduce the training cost of networks on manifolds is to use the surjection between the orthogonal group and its Lie algebra to transform the geometric optimization on the manifold into an optimization problem in Euclidean space. Summary of the invention
[0007] In order to solve the existing bottleneck of ViT, the purpose of the present invention is to provide a method for improving the accuracy of image recognition by using an orthogonal visual converter in an image recognition scenario. Starting from alleviating the problem of gradient instability when training a visual converter, the method restricts the three projection matrices (query projection matrix, key projection matrix and value projection matrix) in the self-attention mechanism in the visual converter network to an orthogonal shape, that is, restricts the above three projection matrices from arbitrary matrices to orthogonal matrices. On the one hand, the orthogonal matrix can alleviate the problems of gradient explosion and gradient disappearance, making the training more stable: when training a neural network, chain derivation will lead to power operations (the result of chain derivation involves several powers of the maximum eigenvalue of the parameter matrix, and the projection matrix is also one of the parameters). The maximum eigenvalue of the parameter matrix is rarely exactly 1. If it is greater than 1, it will cause gradient explosion under the action of power operations; if it is less than 1, it will cause gradient disappearance under the action of power operations. The orthogonal matrix has norm preservation (let A represent an orthogonal matrix, X represent input, ||·|| 2represents the norm, then ||AX|| 2 =(AX) T (AX) = X T A T AX=X T X=||X|| 2 , that is, the power operation of the orthogonal matrix disappears in the final result, and its size will not affect the gradient), so making the projection matrix an orthogonal matrix can make the gradient more stable during training. On the other hand, the row vectors or column vectors in the orthogonal matrix are orthogonal to each other, that is, they are independent of each other, which alleviates the problem of feature redundancy and thus the overfitting problem. The stabilization of the gradient and the alleviation of overfitting both help to improve the image recognition accuracy of the orthogonal visual converter. The present invention also utilizes the surjective mapping between the orthogonal group and its Lie algebra to transform the optimization problem on the manifold into the optimization problem on the Euclidean space, thereby effectively reducing the cost of training the visual converter on the manifold and making the training more stable.
[0008] The specific technical solution for achieving the purpose of the present invention is:
[0009] The present invention provides a method for improving image recognition accuracy and alleviating the problem of gradient instability when training a visual converter by using an orthogonal visual converter. The method comprises the following specific steps:
[0010] Step 1: Generate the query matrix, key matrix, and value matrix required for the self-attention layer in the orthogonal visual transformer:
[0011] Step 1.1, declare and initialize the projection matrix in the self-attention layer;
[0012] First declare the size of the projection matrix (or the number of rows and columns), and then initialize the value of the projection matrix (because without a value, it cannot participate in the calculation). The initialization is divided into the following steps: i) extract random numbers from the standard normal distribution N~(0,1) to fill the matrix; ii) perform QR decomposition (orthogonal triangular decomposition) on the matrix: decompose the projection matrix after step i) into the product of two matrices, which are denoted as Q and R, where Q is an orthogonal matrix and R is an upper triangular matrix; iii) apply a sign function to the diagonal elements of the R matrix: elements greater than 0 are mapped to 1 by the function, elements less than 0 are mapped to -1 by the function, and elements equal to 0 are mapped to 0 by the function; iv) multiply the Q matrix and the R matrix after step iii) to obtain a new projection matrix.
[0013] Step 1.2, using the surjective mapping between the orthogonal group and its Lie algebra, orthogonalize the projection matrices used to generate the query matrix, key matrix and value matrix in step 1.1 respectively;
[0014] Step 1.3: In front of the self-attention layer, there is a patch embedding and position embedding layer for the input image, and its output is the input of the self-attention layer. The query matrix, key matrix, and value matrix are generated from the input of the self-attention layer and the orthogonal projection matrix generated in step 1.2. The implementation process can be expressed as:
[0015] Q,K,V=XW Q ,XW K ,XW V .
[0016] Among them, X represents the input of the self-attention layer; W Q , W K , W V They represent the projection matrices for generating the query matrix, key matrix, and value matrix respectively; Q represents the query matrix, K represents the key matrix, and V represents the value matrix.
[0017] Step 2: Calculate self-attention based on the query matrix, key matrix and value matrix (based on the orthogonal projection matrix):
[0018] Step 2.1. Use the query matrix and key matrix to calculate the similarity between every two vectors. The product of the transpose of the query matrix and the key matrix is the similarity between the query vector and the key vector. The principle is: according to the knowledge of vector dot product in mathematics, if two vectors are in the same direction, their dot product is greater than 0; if two vectors are perpendicular, their dot product is equal to 0; if two vectors are in opposite directions, their dot product is less than 0. Therefore, the dot product of two vectors can represent their similarity. The larger the value of the dot product, the more consistent the directions of the two vectors are, that is, the more similar they are. The similarity matrix obtained in this step is to prepare for the self-attention matrix in the next step. The formula of the similarity matrix is expressed as:
[0019] S=QK T
[0020] Where S represents the similarity matrix, Q represents the query matrix, and K T Represents the transpose of the key matrix.
[0021] Step 2.2, obtain the self-attention matrix from the above similarity matrix;
[0022] Let the dimension of the key matrix be d (this dimension is set by the implementer according to actual needs and the specific value is not specified), use Scale the similarity matrix obtained in step 2.1, and then normalize the scaled similarity matrix using the softmax function to obtain the self-attention matrix, whose formula is:
[0023]
[0024] Among them, A represents the self-attention matrix, and S represents the similarity matrix obtained in step 2.1.
[0025] Step 2.3, multiply the self-attention matrix and the value matrix. The product is the output matrix corresponding to the input matrix of the self-attention layer. The formula is:
[0026] O=AV
[0027] Among them, O represents the output matrix, A represents the self-attention matrix obtained in step 2.2, and V represents the value matrix.
[0028] Step 3: The above attention mechanism can be extended to multiple heads (based on orthogonal projection matrix):
[0029] The multi-head splices together the outputs of several single-head self-attention layers, which plays a role in integration and alleviating overfitting.
[0030] Step 3.1, calculate the output of the single-head self-attention layer;
[0031] Let X be the input of the self-attention layer, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and the formula for the attention matrix of a single self-attention head is expressed as:
[0032]
[0033] Where d is the dimension of the key matrix, K T is the transpose of the key matrix.
[0034] Step 3.2: Calculate the multi-head self-attention matrix.
[0035] Assuming the number of self-attention heads is n, the multi-head attention matrix is composed of n single-head attention matrices, and its formula is expressed as:
[0036] MultiHead(X)=[head 1 ;···;head n ]
[0037] Among them, head i represents the i-th single self-attention head.
[0038] Step 4: Train the orthogonal visual converter model (based on the orthogonal projection matrix):
[0039] Step 4.1, calculate the predicted probability of each category of the image;
[0040] The self-attention layer outputs MultiHead(X), which can be followed by one or more fully connected layers. The output of the last fully connected layer is the predicted probability of each category of the image recognition task.
[0041] Step 4.2, calculate the loss function of each category prediction probability and the true category;
[0042] The loss function is a function that measures the degree of deviation between the predicted value and the true value. There are many ways to implement it, and the common ones are the square loss function and the cross entropy loss function. The present invention uses the cross-entropy loss function, which is expressed as follows:
[0043]
[0044] Among them, C represents the cross entropy loss function, n represents the number of samples for one training (in the image recognition scenario, one picture is also one sample), and y i represents the actual value of the i-th sample, a i Represents the predicted value of the i-th sample.
[0045] Step 4.3, calculate the gradient of the loss function with respect to the network weights of each layer;
[0046] The gradient of the loss function to each layer of network weight is calculated according to the chain rule. The orthogonal visual converter proposed in this invention belongs to the category of neural network. The essence of neural network is a composite function: Assume that the neural network has k layers, let L i represents the i-th layer network, L i The output is L i+1 Let x represent the input of the first layer of the neural network and a represent the output of the last layer of the neural network, then a can be expressed as:
[0047] a=L k (L k-1 (...L1(x))),
[0048] The above formula represents a composite function. Therefore, we need to use the chain rule of composite functions to calculate the gradient. Let's take a simple example to illustrate: the composite function g(f(x)) is composed of two functions. Let u = g(y), y = f(x). The gradient of the composite function is:
[0049]
[0050] This conclusion can be extended to the case of any finite number of functions being combined. The gradient of the composite function is the product of the gradients of these finite functions at corresponding points. Since it is linked together like a chain, it is called the chain rule.
[0051] Step 4.4, back propagate the gradient from back to front, and use the gradient calculated in the previous step to update the weights of each layer of the network;
[0052] Step 4.5: Repeat steps 4.1 to 4.4 until the loss function converges to obtain the trained orthogonal visual converter model. The projection matrix remains orthogonal after each round of update.
[0053] Step 5: Use the orthogonal visual transformer model obtained in step 4 to perform image recognition.
[0054] In the present invention, the projection matrix orthogonalization adopts the surjective mapping between the skew-symmetric matrix and the orthogonal group, which ensures that the projection matrix can be converted into a skew-symmetric matrix and then into an orthogonal matrix. The orthogonal group is composed of orthogonal matrices and is formally defined as:
[0055] O(n)={B∈R n×n |B T B = E},
[0056] Among them, R represents a real number, R n×n represents a real matrix with n rows and columns; E represents the identity matrix; B represents a real orthogonal matrix of size n×n, satisfying the condition that "the product of its transpose and itself is the identity matrix". An orthogonal group is a type of Lie group. Mathematically, the tangent space at the identity element of a Lie group is called the Lie algebra of the Lie group. The Lie algebra of an orthogonal group is a skew-symmetric matrix, which is formally defined as:
[0057] so(n)={A∈R n×n |A+A T =0},
[0058] Among them, 0 represents a zero matrix (all elements in the matrix are 0), and A represents a real skew-symmetric matrix of size n×n, which satisfies the condition that "the sum of itself and its transpose is a zero matrix".
[0059] Specifically, the orthogonalization method needs to first transform the matrix into a skew-symmetric matrix, and the transformation method is as follows:
[0060] SkewSymmetric(W) = WW T ;
[0061] Then use the skew-symmetric matrix as a transition to generate an orthogonal matrix. The generation method is as follows:
[0062] Orthogonalize(W) = 2(E + skewSymmetric(W)) -1 -E;
[0063] The superscript “-1” indicates an inversion operation, and E indicates the identity matrix.
[0064] The orthogonalized projection matrix corresponds to the skew-symmetric matrix one by one, and the two can be transformed into each other, that is, each orthogonal matrix has a skew-symmetric matrix corresponding to it, so that the optimization problem on the orthogonal shape can be transformed into the optimization problem of Euclidean space. The skew-symmetric matrix corresponding to the orthogonal matrix can be expressed as:
[0065] W_skew=2(E+W_orth) -1 -E;
[0066] Among them, W_skew represents a skew-symmetric matrix, and W_orth represents an orthogonal matrix.
[0067] The training process of the orthogonal visual transformer can be transformed into an optimization problem in Euclidean space. Let x denote the input and θ denote the parameter. Let θ B Representing the parameters on the orthogonal manifold G, the optimization problem on the manifold can be abstracted as:
[0068]
[0069] Let θ A Representing the parameters in Euclidean space, the optimization problem in Euclidean space can be abstracted as:
[0070]
[0071] For every orthogonal matrix there is a skew-symmetric matrix corresponding to it, and the skew-symmetric matrix space is homeomorphic to the vector space (the finite-dimensional vector space with defined norms and inner products is the Euclidean space). B There is a θ A Correspondingly, let The optimal solution is exist and Correspondingly, this Just The optimal solution of . Similarly, each θ A There is also a θ B Correspondingly, if we can find The optimal solution of Therefore, the geometric optimization problem of orthogonal matrices can be transformed into an optimization problem in Euclidean space, avoiding the complex operation of updating parameters along geodesics on orthogonal polygons.
[0072] The orthogonal visual converter model described in the present invention will not produce a saddle point, that is, the orthogonal visual converter model can converge to the optimal point.
[0073] The present invention also provides an application of the above method in improving the prediction accuracy of image recognition.
[0074] The beneficial effects of the present invention include: 1. The projection matrix in the self-attention is restricted to an orthogonal manifold, and the optimization problem on the manifold is converted into an optimization problem in Euclidean space by using surjective mapping, thereby reducing the training cost (solving the optimization problem on the manifold requires orthogonal projection and backoff operations with large computational complexity, while the above operations can be omitted by converting it into an optimization problem in Euclidean space, thereby reducing the training cost). 2. Improving the image prediction accuracy (compared with different baseline visual converters, the degree of improvement in the image prediction accuracy of the method of the present invention is different. Compared with the most classic BaseViT, the method of the present invention improves the image prediction accuracy by 2.2%; compared with the variants DeepViT and CaiT (both of which are still non-orthogonal visual converters), the method of the present invention improves the image prediction accuracy by 27.2% and 87.5%). BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 This is a flow chart related to the method of the present invention. The start and end are represented by ellipses, the steps are represented by rectangles, and the judgment is represented by diamonds (at the end of each training, it is necessary to judge whether the loss function converges. If it converges, the training ends; otherwise, the gradient is calculated, the parameters are updated, and a new round of training is performed. In each training, the projection matrix remains orthogonal).
[0076] Figure 2 is a result diagram of an embodiment of the present invention. The horizontal axis of the result diagram represents the number of training rounds, and the vertical axis represents the inference accuracy (in percentage). Figure 2 Each sub-graph of has two curves, representing an orthogonal visual converter and a non-orthogonal visual converter in contrast. The legend will explain which curve corresponds to which visual converter, and the present invention also makes a distinction in naming, and the O prefix represents an orthogonal visual converter (orthogonal). DETAILED DESCRIPTION
[0077] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.
[0078] The present invention proposes a method for alleviating the problem of gradient instability by using an orthogonal visual converter for image recognition scenarios, which can effectively improve the image prediction accuracy and accelerate the convergence speed. The present invention first orthogonalizes the three projection matrices involved in the self-attention mechanism. In each round of training of the orthogonal visual converter, the query projection matrix, the key projection matrix, and the value projection matrix are updated in the direction of decreasing the loss function, but still remain orthogonal. Until the end of the training, an orthogonal visual converter model with high image recognition accuracy and robustness is obtained. The orthogonal visual converter model structure in the present invention is not much different from the non-orthogonal or traditional visual converter model structure. The difference is that the present invention requires the projection matrix in the model to reside on an orthogonal waveform, and subsequent steps such as the self-attention layer and training are based on the orthogonal projection matrix; while the traditional visual converter does not make special requirements for the projection matrix.
[0079] The present invention obtains an orthogonal visual converter model with high image recognition accuracy and robustness by restricting the parameter matrix that generates the query matrix, key matrix and value matrix in the self-attention mechanism to an orthogonal shape. In each round of training of the orthogonal visual converter, the parameter matrix is updated according to the gradient, but still remains orthogonal. The present invention constructs an orthogonal matrix through a surjective mapping between an orthogonal group and its Lie algebra, so the training of the orthogonal visual converter is an optimization problem in Euclidean space rather than a complex geometric optimization. The method described in the present invention can improve the reasoning accuracy of traditional visual converters in image recognition scenarios while having a certain degree of robustness.
[0080] The present invention comprises the following steps:
[0081] Step 1: Generate the query matrix, key matrix, and value matrix required for the self-attention layer in the orthogonal visual transformer:
[0082] 1) Declare and initialize the projection matrix in the self-attention layer;
[0083] 2) orthogonalizing the above projection matrix that generates the query matrix, the key matrix, and the value matrix by using the surjective mapping between the orthogonal group and its Lie algebra;
[0084] 3) The patch embedding and position embedding layers for the input image are set before the self-attention layer, and their output is the input of the self-attention layer. The query matrix, key matrix and value matrix are generated from the input of the self-attention layer and the orthogonal projection matrix generated above.
[0085] Step 2: Calculate self-attention based on the query matrix, key matrix and value matrix (based on the orthogonal projection matrix):
[0086] 1) Using the obtained query matrix and key matrix, the similarity between every two vectors is calculated to obtain a similarity matrix;
[0087] The product of the transposed query matrix and the key matrix is the similarity between the query vector and the key vector.
[0088] 2) Obtain the self-attention matrix from the above similarity matrix;
[0089] Let the dimension of the key matrix be d, and use Scale the similarity matrix, and then normalize the scaled similarity matrix using the softmax function to obtain the self-attention matrix.
[0090] 3) Use the self-attention matrix and the value matrix to calculate the output matrix corresponding to the input matrix.
[0091] The product of the self-attention matrix and the value matrix is the output matrix.
[0092] Step 3: The above attention mechanism can be extended to multiple self-attention heads (based on orthogonal projection matrix):
[0093] 1) Calculate the attention matrix of a single self-attention head;
[0094] Let X be the input of the self-attention layer, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and the formula for the attention matrix of a single self-attention head is expressed as:
[0095]
[0096] Where d is the dimension of the key matrix, K T is the transpose of the key matrix.
[0097] 2) Calculate the attention matrix of multiple self-attention heads.
[0098] Assuming the number of self-attention heads is n, the multi-head attention matrix is composed of n single-head attention matrices, and its formula is expressed as:
[0099] MultiHead(X)=[head 1 ;···;head n ]
[0100] Among them, head i represents the i-th single self-attention head.
[0101] Step 4: Train the orthogonal visual converter (based on the orthogonal projection matrix):
[0102] The output obtained by the self-attention mechanism can be followed by one or more fully connected layers. The output of the last fully connected layer is the predicted probability of each category of the image recognition task. Calculate the loss function of the predicted probability and the true category of each category, and find the gradient of the loss function with respect to the weight of each layer of the network. Backpropagate the gradient from back to front, and use the gradient to update the weight of each layer of the network. Repeat until the value of the loss function converges. The projection matrix remains orthogonal after each round of update.
[0103] The steps specifically include:
[0104] (1) Projection matrix orthogonal strategy: Any matrix can be transformed into a skew-symmetric matrix, and a skew-symmetric matrix can be transformed into an orthogonal matrix. Any matrix W satisfies: The difference between the matrix and its transposed matrix, WW T is a skew-symmetric matrix. Let E be the unit matrix. Any skew-symmetric matrix W_skew satisfies: 2(E+W_skew) -1 -E is an orthogonal matrix. Let W Q represents the projection matrix that generates the query matrix, W K represents the projection matrix that generates the key matrix, W V represents the projection matrix of the generated value matrix, W Q , W K , W V They are all trainable linear projection matrices. They are first randomly initialized, and then converted into orthogonal matrices according to the above steps, using skew-symmetric matrices as transitions.
[0105] (2) Query matrix, key matrix and value matrix generation strategy: The visual transformer first embeds the input image into a fixed-size sequence, and then embeds their position information to obtain patch embedding and position embedding. The input image after patch embedding and position embedding is the input X of self-attention. Let Q represent the query matrix, K represent the key matrix, and V represent the value matrix. After projection transformation, the generation process of the query matrix, key matrix and value matrix is as follows:
[0106] Q,K,V=XW Q ,XW K ,XW V .
[0107] In one training cycle of the Orthogonal Visual Transformer, the projection matrix that generates the query matrix, key matrix, and value matrix is first orthogonalized, and then the query matrix, key matrix, and value matrix are generated and self-attention is calculated. The self-attention layer can be followed by a fully connected layer. The last fully connected layer outputs the predicted probability of each image category, calculates the loss function and back-propagates, and updates the parameters of each network layer. The above process is repeated until the Orthogonal Visual Transformer converges.
[0108] The specific algorithm is as follows:
[0109]
[0110]
[0111] The present invention alleviates the problem of gradient instability in the self-attention layer in the visual converter, and the gradient instability refers to the gradient explosion and gradient vanishing problems. When training a neural network, chain derivation will lead to power operations (the result of chain derivation involves several powers of the maximum eigenvalue of the parameter matrix, and the projection matrix is also one of the parameters). The maximum eigenvalue of the parameter matrix is rarely exactly 1. If it is greater than 1, it will cause gradient explosion under the action of power operations; if it is less than 1, it will cause gradient vanishing under the action of power operations. The orthogonal matrix has the property of norm preservation (let A represent the orthogonal matrix, X represent the input, ||·|| 2 represents the norm, then ||AX|| 2 =(AX) T (AX) = X T A T AX=X T X=||X|| 2 , that is, the power operation of the orthogonal matrix disappears in the final result, and its size will not affect the gradient). Therefore, making the projection matrix an orthogonal matrix can make the gradient more stable during training, and restricting the projection matrix in the self-attention to an orthogonal shape improves the accuracy of identifying image categories.
[0112] Example
[0113] The method of the present invention focuses on the scene of image recognition. As a specific embodiment, the process is as follows:
[0114] 1. The Yale face dataset is selected as the experimental dataset of the present invention. The training set contains 2314 pictures, the test set contains 1874 pictures, and both the training set and the test set contain 38 categories.
[0115] 2. In order to increase data and improve the generalization ability of the method, this embodiment uses random rotation, cropping, and horizontal flipping strategies to expand the data set.
[0116] 3. First, set the image size to 32×32, process each image into a patch sequence of 4×4 patch size, and embed the position information to obtain the input features of the self-attention layer.
[0117] 4. Orthogonalize the query projection matrix, key projection matrix and value projection matrix, and use these three projection matrices to transform the input features into query matrix, key matrix and value matrix respectively.
[0118] 5. Use the query matrix and key matrix to obtain the self-similarity of the query vector and the key vector, apply the normalized and scaled self-similarity matrix to the value matrix, and get the output of the self-attention layer.
[0119] 6. Set the number of self-attention heads and concatenate the output of each single attention head to get multi-head self-attention.
[0120] 7. Set the orthogonal visual converter to train for 100 rounds. In each round of training, first pass the output of the self-attention layer to the fully connected layer to predict the probability of the image belonging to each category. Calculate the loss function of the predicted probability and the true category of the image, find the gradient backpropagation, and use the gradient to update the parameters of each network layer, including the three orthogonal projection matrices. The projection matrix remains orthogonal after each round of update.
[0121] 8. After every ten rounds of parameter updates, the orthogonal visual converter model is tested for inference accuracy. The images in the test set are used to record the ratio of the number of correctly inferred images to the total number of images in the test set as the inference accuracy. The test results are recorded and visualized as follows: Figure 1 .
[0122] As a comparison of experimental results, the more mainstream visual converter algorithms (BaseViT, DeepViT, CaiT) are used as benchmarks. The methods of the present invention are O-BaseViT, O-DeepViT, and O-CaiT. Figure 1 The experimental results show that: compared with the current mainstream visual converter algorithm, the method proposed in the present invention is superior to the traditional non-orthogonal visual converter algorithm. Specifically: 1. At the same number of training rounds, the method proposed in the present invention has a higher recognition accuracy than the traditional visual converter algorithm (from the experimental result graph, it can be seen that at about 100 rounds, the reasoning accuracy obtained by the method proposed in the present invention is higher than that of several benchmark algorithms, and the image prediction accuracy from left to right has increased by 2.2%, 27.2%, and 87.5% respectively); 2. The method proposed in the present invention converges faster than the traditional method during training (Note: from the experimental result graph, it can be seen that when the reasoning accuracy of the method proposed in the present invention is close to 100%, the number of training rounds required is less than that of several benchmark algorithms)
[0123] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.
Claims
1. A method for improving image recognition accuracy using an orthogonal visual converter, characterized in that: The method comprises the following specific steps: Step 1: Generate the query matrix, key matrix, and value matrix required for the self-attention layer in the orthogonal visual transformer; Step 2: Based on the orthogonal projection matrix, calculate the self-attention according to the query matrix, key matrix and value matrix; The step 2 further comprises: Step 2.1, characterize the similarity between the query vector and the key vector by calculating the product of the query matrix and the transposed key matrix; Step 2.2, let the dimension of the key matrix be d, and use Scale the similarity matrix obtained in step 2.1, and then normalize the scaled similarity matrix using the softmax function to obtain the self-attention matrix; Step 2.3, multiply the self-attention matrix and the value matrix to calculate the output matrix corresponding to the input matrix of the self-attention layer; Step 3: Based on the orthogonal projection matrix, expand the self-attention mechanism in step 2 to multiple heads; Step 4: Based on the orthogonal projection matrix, train the orthogonal visual converter model; in the step 4, the output MultiHead(X) of the self-attention mechanism is connected to one or more fully connected layers; wherein the output of the last fully connected layer is the predicted probability of each category of the image recognition task; calculate the loss function of the predicted probability and the true category of each category, and calculate the gradient of the loss function to the weight of each layer of the network; back-propagate the gradient from back to front, and use the gradient to update the weight of each layer of the network; repeat until the value of the loss function converges, and obtain the trained orthogonal visual converter model; Step 5: Use the orthogonal visual transformer model obtained in step 4 to perform image recognition.
2. The method according to claim 1, characterized in that Step one further comprises: Step 1.1, declare and initialize the projection matrix in the self-attention layer; Step 1.2, using the surjective mapping between the orthogonal group and its Lie algebra, orthogonalize the projection matrix used to generate the query matrix, key matrix and value matrix in step 1.1; Step 1.3, generate a query matrix, a key matrix and a value matrix from the input of the self-attention layer and the orthogonal projection matrix generated in step 1.2; the input of the self-attention layer comes from the output of the patch embedding and position embedding layer for the input image set before the self-attention layer.
3. The method according to claim 2, characterized in that In step 1.2, the orthogonalization needs to first transform the matrix into a skew-symmetric matrix. The transformation method is as follows: SkewSymmetric(W)=WW T ; Then use the skew-symmetric matrix as a transition to generate an orthogonal matrix. The generation method is as follows: Orthogonalize(W) = 2(E + skewSymmetric(W)) -1 -E; The superscript "-1" indicates an inversion operation, and E indicates the identity matrix.
4. The method according to claim 3, characterized in that The orthogonalized projection matrix corresponds to the skew-symmetric matrix one by one; the skew-symmetric matrix corresponding to the orthogonal matrix is expressed as: W_skew=2(E+W_orth) -1 -E; Among them, W_skew represents a skew-symmetric matrix, and W_orth represents an orthogonal matrix.
5. The method according to claim 1, characterized in that Step three may further include: Step 3.1, calculate the output of the single-head self-attention layer; Let X be the input of the self-attention layer, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and the formula for the attention matrix of a single self-attention head is expressed as: Where d is the dimension of the key matrix, K T is the transpose of the key matrix; Step 3.2, calculate the multi-head self-attention matrix: Assuming the number of self-attention heads is n, the multi-head attention matrix is composed of n single-head attention matrices, and its formula is expressed as: MultiHead(X)=[head 1 ;···;head n ], Among them, head i represents the i-th single self-attention head.
6. The method according to claim 1, characterized in that The orthogonalized visual transformer avoids computationally expensive geometric optimization by converting it into an optimization problem in Euclidean space.
7. The method according to claim 1, characterized in that The orthogonal visual converter model will not produce a saddle point, that is, the orthogonal visual converter model can converge to the optimal point.
8. Application of the method as claimed in any one of claims 1 to 7 in improving the prediction accuracy of image recognition.