A vit model optimization method suitable for face recognition

Abstract

The application discloses a ViT model optimization method suitable for face recognition, which comprises the following steps: performing feature point detection on a face image to obtain a feature point coordinate set; dividing the face image into multiple image blocks, and obtaining a corresponding feature point number matrix based on the multiple image blocks; processing edge elements of the feature point number matrix, and obtaining a weight matrix without zero elements by interpolating the zero element positions of the matrix; inputting the face image and the corresponding weight matrix into a ViT network to obtain the probability that the input face image belongs to each registered user. The application can utilize the weight to reduce the influence of non-face parts in the image on the final face recognition, and has higher convergence speed and accuracy.

Description

Technical Field

[0001] This invention relates to the field of face recognition technology, and in particular to a ViT model optimization method suitable for face recognition. Background Technology

[0002] Facial recognition technology is widely used in scenarios such as facial recognition payment, criminal investigation, and facial modeling. This demonstrates the profound research significance of facial recognition technology. In June 2017, Google published "Attention Is All You Need" at NIPS 2017, proposing the Transformer for application in Natural Language Processing (NLP). The main framework of the Transformer model is an encoder-decoder structure, completely removing the convolutional neural network (RNN) structure, and connected by attention mechanisms and fully connected layers. Inspired by the Transformer model in NLP, Google published "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" at ICLR 2021 in November 2020, proposing to directly apply the Transformer to the field of Computer Vision (CV), proving its feasibility in CV, and calling it the Vision Transformer (ViT).

[0003] The ViT model consists of three main modules: Linear Projection of Flattened Patches (Embedding), Transformer Encoder, and MLP Head. The Linear Projection of Flattened Patches layer uses convolutional operations to divide the image into patches, transforming these patches into vectors and combining them with positional information before inputting them into the Transformer Encoder layer. The Transformer Encoder layer is composed of L stacked Encoder Blocks, each consisting of Layer Norm, Multi-Head Attention, Dropout, and an MLP Block. The MLP Head layer is used for data classification.

[0004] The ViT model's Multi-Head Attention includes a self-attention mechanism. In this mechanism, image patch vectors are used as the Query and Key-Value inputs, respectively. The Query and Key are used as inputs to calculate the attention function value, which is then weighted and output with the corresponding Value. The ViT model treats each small image patch in a face image the same, only observing the similarity between patches. However, in specific problems like face recognition, different image patches can have varying degrees of importance; patches containing the face are more important than background patches. Therefore, facial feature points are introduced to enhance the differentiation of importance between image patches. Facial feature point detection is commonly used to enhance face recognition capabilities, facial expression recognition, AI face swapping, and beautification. A face has 68 different feature points used to describe the facial contour, the positions of the eyes, nose, and mouth. The index values ​​of the feature points used for each part are fixed; for example, points 1-17 describe the facial contour, and points 18-27 describe the upper eyebrow, etc.

[0005] ViT can be used for classifying any image, but due to the different feature distributions of images, the classification accuracy of ViT will vary. If the original ViT model is used to process face recognition problems, the accuracy will not meet expectations. Inspired by traditional face recognition algorithms, facial feature point detection methods can be combined with the ViT model to improve the recognition accuracy of the ViT model in face tasks. Summary of the Invention

[0006] To address this, the present invention provides a ViT model optimization method suitable for face recognition, which can utilize weights to reduce the influence of non-face parts in the image on the final face recognition, resulting in higher convergence speed and accuracy.

[0007] This invention discloses a ViT model optimization method suitable for face recognition, which includes the following steps:

[0008] Feature point detection is performed on the face image to obtain the feature point coordinate set;

[0009] The face image is divided into multiple image blocks, and a matrix of the number of feature points corresponding to each image block is obtained.

[0010] The edge elements of the feature point count matrix are processed, and a weight matrix without zero elements is obtained by interpolating the positions of the zero elements of the matrix.

[0011] The face image and its corresponding weight matrix are input into the ViT network to obtain the probability that the input face image belongs to each registered user.

[0012] Furthermore, the step of performing feature point detection on the face image to obtain a set of feature point coordinates includes:

[0013] Let Image be the image of the i-th person's face. i For Image i Perform feature point detection to obtain an Image i The coordinate set of each facial feature point [[x0, y0], [x1, y1]...[x n y n ]] j ; where, [x j y j ] represents the coordinates of the j-th feature point.

[0014] Furthermore, the face image is divided into multiple image blocks, and based on the position information of the image blocks, a matrix of the number of corresponding feature points is obtained, including:

[0015] Image i Divide the image into K equal-sized blocks, and calculate the x-th block for each block. j The number of facial feature points in the data (num) j Finally, it is arranged according to the image block in the i-th face image. i The positions in the matrix form the feature point count matrix M. i ; where j ranges from 1 to K.

[0016] Furthermore, the edge elements of the feature point count matrix are processed by interpolating the positions of the zero elements in the matrix to obtain a weight matrix without zero elements, including:

[0017] The zero elements at the edge of the matrix are assigned a value of 0.1. All zero element positions in the matrix are considered empty. Interpolation is then performed on the positions of the zero elements in the matrix. That is, the feature point count matrix M is first processed separately. i Linear interpolation of the row vectors yields the matrix MA. i Then, linear interpolation is performed on its column vectors to obtain the matrix MB. i Finally, take matrix MA. i and matrix MB i The average value is the matrix MC. i ;

[0018] MC matrix i All elements less than 1 are assigned a value of 0, and all elements greater than or equal to 1 are assigned a value of 1, thus obtaining the final weight matrix, denoted as WM. i .

[0019] Further, the step of inputting the face image and its corresponding weight matrix into the ViT network to obtain the probability that the face image belongs to each registered user includes:

[0020] The face image and its corresponding weight matrix are input into the ViT network. In the network, the weight matrix is ​​multiplied by the attention function value in the attention mechanism to obtain new weights. The output of the ViT network is a vector to represent the probability of each registered user.

[0021] Furthermore, the face image and its corresponding weight matrix are input into the ViT network to obtain the probability that the face image belongs to each registered user, specifically including:

[0022] The face image is divided into blocks. i Input the Linear Projection of Flattened Patches module; the output is z0.

[0023] Connect z0 and the weight matrix WM i Input the Transformer Encoder module, the MSA module will output WM i Multiply by the attention function value:

[0024] The final output of the ViT network is z. l ′=MSA(LN(z l-1 ))+z l-1 , where 1≤l≤L.

[0025] Furthermore, the step of dividing the face image into blocks... i Input the Linear Projection of Flattened Patches module, output z0, which includes:

[0026] Image of face i With weight matrix MD i When the image is packaged and input into the ViT network, it first enters the Linear Projection of Flattened Patches module, which divides the image into N image blocks of size P*P*C x. j , divide each image patch x j The input is transformed into a vector of length D by a fully connected layer E, and then compared with the bias term x. class Connect, and then add the position information E from the Embedding layer. pos The output of the Linear Projection of Flattened Patches module is z0:

[0027] z0 = [x classx1E; x2E; x3E; ...x N E]+E pos .

[0028] Furthermore, the step of combining z0 and the weight matrix WM i Input the Transformer Encoder module, the MSA module will output WM i Multiplying by the attention function value is expressed by the following formula:

[0029]

[0030] output = WM i *Attention(Q, K, V)

[0031] Where Attention(Q, K, V) represents the weights of the value, Q is the query, K is the key, and V is the value. The values ​​of Q, K, and V are the same. The attention function values ​​of Q and K are... The weights are processed by the softmax function so that the sum of the weights is 1, and this sum is used as the weight of V. The weighted output is the value of Attention(Q, K, V).

[0032] Furthermore, the Transformer Encoder module includes L linearly connected Transformer blocks, each Transformer block including an MSA Block and an MLP Block. The MSA Block includes a multi-head self-attention mechanism, Layer-Norm, and residual connections.

[0033] MLP Block includes feedforward network, Layer-Norm, and residual connection;

[0034] The output of the Transformer Encoder module is z. l :

[0035] z l =MLP(LN(z′) l ))+z l

[0036] Where l = 1, ..., L.

[0037] Furthermore, the output value z of the Transformer Encoder module l The image is then entered into the MLP Head layer for classification to determine the probability that a face image belongs to each registered user.

[0038] Due to the adoption of the above technical solution, the present invention has the following advantages: the method proposed in this invention can use weights to reduce the influence of non-face parts in the image on the final face recognition, and has higher convergence speed and accuracy. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0040] Figure 1 A schematic diagram of 68 feature points of a face;

[0041] Figure 2 This is a flowchart of the weight calculation process for this invention;

[0042] Figure 3 This is a network structure diagram of the method of the present invention;

[0043] Figure 4 The graph shows the accuracy results for different control groups;

[0044] Figure 5 Numerical plots of the loss functions for different control groups;

[0045] Figure 6 This is a comparison chart of the accuracy of the second training set and the test set;

[0046] Figure 7 This is a flowchart illustrating a ViT model optimization method suitable for face recognition. Detailed Implementation

[0047] The present invention will be further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0048] The purpose of this invention is to optimize the ViT model to make it more suitable for face recognition. Observing the distribution of facial feature points, it is found that the feature points in the eye, nose, and mouth regions are densely distributed, and these organ parts are the most important regions for face recognition. Therefore, the importance of each block can be simulated by the number of feature points in the image block; the more feature points a block contains, the more important it is. In the ViT model, feature point detection is first performed on the face image, and the coordinates of the feature points are returned. Then, the number of feature points in each block is calculated according to the size of the image blocks, and these are used to form an initial weight matrix based on the position of the image blocks in the face image. There is a problem with the initial weight matrix: because face images have local correlation, i.e., the surrounding areas of important parts are also important, and the values ​​of adjacent elements in the initial weight matrix differ greatly (e.g., the mouth and cheek are adjacent, the mouth area has a dense distribution of feature points, while the cheek area has no feature points), therefore, interpolation of the initial weight matrix is ​​required.

[0049] The non-zero elements of the initial weight matrix are used as input, and the zero elements are considered as interpolation elements. Since the positions of the non-zero elements are not fixed and vary with the face position, conventional two-dimensional interpolation methods cannot be used. This invention employs a method similar to bilinear interpolation, performing linear interpolation on the initial weight matrix both horizontally and vertically, and then averaging the two results to obtain the final weight matrix. Because linear interpolation is required, the edge non-zero elements of the initial weight matrix cannot be empty and are set to very small values, such as 0.1. At this point, because the face is a convex hull shape, and the weights of the image patches containing the face contour are all greater than or equal to 1, and the matrix edges have very small values, the weights of the face region are all greater than or equal to 1, while the weights of other regions are all less than any weight within the face region. Finally, this is multiplied by the corresponding weight in the attention mechanism to make the image patches in the face region more important.

[0050] See Figure 7 This invention provides an embodiment of a ViT model optimization method suitable for face recognition, which includes the following steps:

[0051] Feature point detection is performed on the face image to obtain the feature point coordinate set;

[0052] The face image is divided into multiple image blocks, and a matrix of the number of feature points corresponding to each image block is obtained.

[0053] The edge elements of the feature point count matrix are processed, and a weight matrix without zero elements is obtained by interpolating the positions of the zero elements of the matrix.

[0054] The face image and its corresponding weight matrix are input into the ViT network to obtain the probability that the input face image belongs to each registered user.

[0055] In this embodiment, the step of detecting feature points in a face image to obtain a set of feature point coordinates includes:

[0056] Let Imagei be the face image of the i-th person. i Perform feature point detection to obtain an Image i The coordinate set of each facial feature point [[x0, y0], [x1, y1]...[x n y n ]] i ; where, [x j y j ] represents the coordinates of the j-th feature point.

[0057] In this embodiment, the face image is divided into multiple image blocks, and based on the position information of the image blocks, a matrix of the number of feature points corresponding to each block is obtained, including:

[0058] Image i Divide the image into K equal-sized blocks, and calculate the x-th block for each block. j The number of facial feature points in the data (num) j Finally, it is arranged according to the image block in the i-th face image. i The positions in the matrix form the feature point count matrix M. i ; where j ranges from 1 to K.

[0059] In this embodiment, the edge elements of the feature point count matrix are processed by interpolating the positions of the zero elements of the matrix to obtain a weight matrix without zero elements, including:

[0060] The zero elements at the edge of the matrix are assigned a value of 0.1. All zero element positions in the matrix are considered empty. Interpolation is then performed on the positions of the zero elements in the matrix. That is, the feature point count matrix M is first processed separately. i Linear interpolation of the row vectors yields the matrix MA. i Then, linear interpolation is performed on its column vectors to obtain the matrix MB. i Finally, take matrix MA. i and matrix MB i The average value is the matrix MC. i ;

[0061] MC matrix i All elements less than 1 are assigned a value of 0, and all elements greater than or equal to 1 are assigned a value of 1, thus obtaining the final weight matrix, denoted as WM. i .

[0062] In this embodiment, the step of inputting the face image and its corresponding weight matrix into the ViT network to obtain the probability that the face image belongs to each registered user includes:

[0063] The face image and its corresponding weight matrix are input into the ViT network. In the network, the weight matrix is ​​multiplied by the attention function value in the attention mechanism to obtain new weights. The output of the ViT network is a vector to represent the probability of each registered user.

[0064] In this embodiment, the face image and its corresponding weight matrix are input into the ViT network to obtain the probability that the face image belongs to each registered user, specifically including:

[0065] The face image is divided into blocks. i Input the Linear Projection of Flattened Patches module; the output is z0.

[0066] Connect z0 and the weight matrix WM i Input the Transformer Encoder module, the MSA module will output WM i Multiply by the attention function value:

[0067] The final output of the ViT network is z. l =MSA(LN(z) l-1 ))+z l-1 , where 1≤l≤L.

[0068] In this embodiment, the face image is divided into blocks. i Input the Linear Projection of Flattened Patches module, output z0, which includes:

[0069] Image of face i With weight matrix MD i When the image is packaged and input into the ViT network, it first enters the Linear Projection of Flattened Patches module, which divides the image into N image blocks of size P*P*C x. j , divide each image patch x j The input is transformed into a vector of length D by a fully connected layer E, and then compared with the bias term x. class Connect, and then add the position information E from the Embedding layer. pos The output of the Linear Projection of Flattened Patches module is z0:

[0070] z0 = [x class x1E; x2E; x3E; ...x N E]+E pos .

[0071] In this embodiment, the step of combining z0 and the weight matrix WM i Input the Transformer Encoder module, the MSA module will output WM i Multiplying by the attention function value is expressed by the following formula:

[0072]

[0073] output = WM i *Attention(Q, K, V)

[0074] Here, Attention(Q, K, V) represents the weights of the value. Self-attention is used here. Q is the query, K is the key, and V is the value. In this method, the values ​​of Q, K, and V are the same, but their uses are different. The attention function values ​​of Q and K are calculated. Here, the attention function is... To ensure that the weighted sum of the weights is 1, the weights are processed by the softmax function. The sum of these values ​​is 1, which is used as the weight of V. The weighted output is the value of Attention(Q, K, V).

[0075] In this embodiment, the Transformer Encoder module includes L linearly connected Transformer blocks. Each Transformer block includes an MSA Block and an MLP Block. The MSA Block includes a multi-head self-attention mechanism, Layer-Norm, and residual connections.

[0076] MLP Block includes feedforward network, Layer-Norm, and residual connection;

[0077] The output of the Transformer Encoder module is z. l :

[0078] z l =MLP(LN(z′) l ))+z l

[0079] Where l = 1, ..., L.

[0080] In this embodiment, the output value z of the Transformer Encoder module l The image is then entered into the MLP Head layer for classification to determine the probability that a face image belongs to each registered user.

[0081] See Figures 1 to 3This invention presents experimental results regarding accuracy. The specific conditions for the simulation experiment are as follows: the training set includes 100,000 face images of 2000 classes, with a batch size of 128. The image size input to the network is 224*224*3, the image block size is 16, the Transformer Encoder repeats the Transformer Block 12 times, and the MSA Block uses a 12-head attention mechanism.

[0082] Figure 4 This figure shows the accuracy comparison curves of the method of the present invention under the specific simulation conditions described above, with different processing methods for the weight matrix. The horizontal axis represents the number of iterations (epochs), and the vertical axis represents the accuracy. Four control groups were set. The first group is the original ViT model; the second group has the weight matrix edges set to 0.1, and interpolation is performed with four decimal places retained; the third group has the weight matrix edges set to 0.1, and then the matrix is ​​multiplied by the cube, retaining four decimal places; the fourth group has the weight matrix edges set to 0.1, and a 0-1 matrix is ​​created where matrix elements less than 1 are assigned a value of 0, otherwise assigned a value of 1.

[0083] from Figure 4 As can be seen, the accuracy and convergence speed of the method of this invention are significantly improved under this dataset, and the second group has the best effect, with an accuracy improvement of nearly 10 percentage points compared with the first group.

[0084] Figure 5 The curves show the comparison of loss function values ​​under different weight matrix processing methods in the specific simulation conditions described above. The loss function is the cross-entropy loss function in a multi-classification task.

[0085]

[0086] The horizontal axis in the graph represents the number of iterations (epochs), and the vertical axis represents the loss function value. Two control groups are set up. The first group is the original ViT model; the second group is a 0-1 matrix with the weight matrix edges set to 0.1. If a matrix element is less than 1, it is assigned a value of 0; otherwise, it is assigned a value of 1.

[0087] from Figure 5 It can be seen that, under this dataset, the loss value of the method of the present invention decreases significantly at the beginning of training, indicating that the learning rate is appropriate and gradient descent is being performed. After learning to a certain stage, the loss curve tends to stabilize and the loss change is not as obvious as at the beginning.

[0088] Figure 6 To compare the accuracy of the training and verification processes in this invention, the weight matrix edges are set to 0.1. If a matrix element is less than 1, it is assigned a value of 0; otherwise, it is assigned a value of 1, forming a 01 matrix.

[0089] from Figure 6 As can be seen, under this dataset, the loss function value of the validation set of the method of this invention is smaller than that of the training set, and the convergence speed is faster, indicating that there is no overfitting phenomenon, and this method is suitable for processing face recognition.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A ViT model optimization method suitable for face recognition, characterized in that, The method comprises the following steps: detecting feature points of a face image to obtain a feature point coordinate set; dividing the face image into a plurality of image blocks, and obtaining a corresponding feature point number matrix based on the plurality of image blocks; processing edge elements of the feature point number matrix, and obtaining a weight matrix without zero elements by interpolating zero element positions of the matrix; inputting the face image and the corresponding weight matrix into a ViT network to obtain a probability that the input face image belongs to each registered user; the feature point detection of the face image to obtain the feature point coordinate set comprises: Let Image be the image of the i-th person's face. i For Image i Perform feature point detection to obtain an Image i The coordinate set of each facial feature point [[x0,y0],[x1,y1]…[x n ,y n ]] i ; where, [x j ,y j ] represents the coordinates of the j-th feature point; the division of the face image into the plurality of image blocks, and the obtaining of the corresponding feature point number matrix based on position information of the image blocks comprises: Image i is divided into K image blocks of the same size, and the number of feature points num j in the jth image block x j is calculated respectively, and finally the number of feature points is arranged in the i-th face image Image i according to the position of the image block to form a feature point matrix M i ; wherein the value range of j is 1 to K; the processing of the edge elements of the feature point number matrix, and the obtaining of the weight matrix without zero elements by interpolating the zero element positions of the matrix comprises: The matrix edge zero element position is assigned to a first preset value, all zero element positions in the matrix are regarded as vacancies, and interpolation is performed on the positions of the zero elements in the matrix, that is, first, linear interpolation is performed on the row vectors of the feature point number matrix M i to obtain a matrix MA i , then linear interpolation is performed on the column vectors to obtain a matrix MB i , and finally, the average value of the matrix MA i and the matrix MB i is taken, that is, a matrix MC i ; the first preset value is a positive number; The matrix MC i is obtained by assigning a third preset value to all elements less than a second preset value and assigning the second preset value to all elements greater than or equal to the second preset value, and is denoted as WM i ; the first preset value is greater than the third preset value and less than the second preset value.

2. The method of claim 1, wherein, the input of the face image and the corresponding weight matrix into the ViT network to obtain the probability that the face image belongs to each registered user comprises: the input of the face image and the corresponding weight matrix into the ViT network, the multiplication of the weight matrix and an attention function value in an attention mechanism in the network to obtain a new weight, and the output of the ViT network as a vector to represent the probability that the face image belongs to each registered user.

3. The method of claim 2, wherein, the input of the face image and the corresponding weight matrix into the ViT network to obtain the probability that the face image belongs to each registered user specifically comprises: Splitting the face image into patches Image i Input: Linear Projection of Flattened Patches module, output is z0; z0 and weight matrix WM i Input Transformer Encoder module, MSA module will WM i Multiply with attention function value: The final output of the ViT network is z l = MSA(LN(z l-1 )) + z l-1 where 1≤l≤L.

4. The method of claim 3, wherein, The human face image is divided into blocks Image i The input Linear Projection of Flattened Patches module outputs z0, including: The face image Image i is multiplied by the weight matrix MD i The input ViT network is packaged, first entering the Linear Projection of Flattened Patches module, dividing the image N into image blocks x of size P*P*C j Each image block x j is sent into a fully connected layer E to be converted into a vector of length D, and a bias term x class is connected, and the position information E after the Embedding layer is added pos The output of the Linear Projection of Flattened Patches module is z0: z0 = [x class ; x1E; x2E; x3E;... x N E] + E pos .

5. The method of claim 4, wherein, The z0 and the weight matrix WM i The input TransformerEncoder module, the MSA module multiplies WM i The multiplication with the attention function value is represented by the following formula: output = WM i *Attention(Q, K, V) Wherein, the Attention(Q, K, V) is the weight value of the value, Q is the query, K is the key, and V is the value, the values of Q, K, and V are the same, and the attention function value of Q and K is The weight is processed by a softmax function, so that the sum of the weights is 1 and the weight is used as the weight of V, and the weighted output is the value of Attention(Q, K, V).

6. The method of claim 5, wherein, the Transformer Encoder module comprises L linearly connected Transformer blocks, each Transformer block comprises an MSA Block and an MLP Block, the MSA Block comprises a multi-head self-attention mechanism, a Layer-Norm, and a residual connection; the MLP Block comprises a feedforward network, a Layer-Norm, and a residual connection; The output of the Transformer Encoder module is z l : z l = MLP(LN(z' (z l ))+ z l wherein, l = 1, …, L.

7. The method of claim 6, wherein, The output value z of the Transformer Encoder module l Enter the MLP Head layer for classification to get the probability that the face image belongs to each registered user.

Images