Multimodal face recognition methods, devices, equipment, and media based on contrastive learning
Through the comparative learning method, the multimodal face features are extracted by combining convolutional neural networks and residual networks, and the feature analysis is performed using 3D convolutional networks, which solves the problem of low face recognition efficiency and accuracy in the existing technology, and realizes the More efficient and accurate facial recognition.
Patent Information
- Application Number
- CN202210520354.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-13
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-05-13
AI Technical Summary
The existing multimodal face recognition method adopts direct splicing when feature fusion, resulting in low facial recognition efficiency and low accuracy.
Using a method based on contrast learning, the features of face plane images and contour images are extracted through convolutional neural networks and residual networks, and then cross-combination is performed using 3D convolutional networks for feature analysis, combining cross-entropy loss function and contrast learning loss function Training the model to improve the accuracy of multimodal feature representation.
It improves the accuracy and efficiency of face recognition, enhances the representation ability of multimodal features, and improves the security and recognition accuracy of the model.
Smart Images

Figure CN114821736B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing and recognition technology, and in particular to a multimodal face recognition method, apparatus, computer equipment and storage medium based on contrastive learning. Background Art
[0002] With the vigorous development of the new generation of artificial intelligence technology, face recognition technology has begun to be widely used in various fields of human society, including access control security, mobile phone facial unlocking, etc. However, while face recognition technology provides convenience for people's daily lives of all kinds, it also brings some security risks. Therefore, multimodal 3D face recognition that combines plane images and depth maps of human facial contour shapes was born.
[0003] However, current methods mostly use a direct splicing approach when fusing multimodal features, which fails to obtain accurate multimodal feature representation and results in problems such as low efficiency and low accuracy in face recognition. Summary of the invention
[0004] Based on this, it is necessary to provide a multimodal face recognition method, device, computer equipment and storage medium based on contrastive learning that can improve the accuracy of face recognition in response to the above technical problems.
[0005] A multimodal face recognition method based on contrastive learning, the method comprising:
[0006] Acquire a face image to be recognized; the face image includes a face plane image and a face contour image;
[0007] Build a face recognition model; the face recognition model includes convolutional neural network, residual network, 3D convolutional network, fully connected layer and SoftMax layer;
[0008] Extract features of the face plane image according to the convolutional neural network to obtain a face plane feature map;
[0009] The residual network is used to extract features from the face contour image to obtain a face contour feature map;
[0010] The face plane feature map and the face contour feature map are cross-combined to obtain multi-modal fusion features;
[0011] The multimodal fusion features are enhanced according to the contrastive learning method to obtain the multimodal feature representation;
[0012] Use 3D convolutional network to perform feature analysis on multimodal feature representation and obtain the probability distribution corresponding to the name;
[0013] A cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, and the pre-constructed face recognition model is trained using the cross entropy loss function and the loss function in contrastive learning to obtain a trained face recognition model;
[0014] Perform face recognition on the face image to be identified according to the trained face recognition model.
[0015] In one embodiment, feature extraction is performed on a face plane image according to a convolutional neural network to obtain a face plane feature map, including:
[0016] A 2D convolutional neural network is used to set a parameter matrix that is less than or equal to the size of the face plane image as the convolution kernel. The face plane image is feature extracted through multiple layers of convolution layers, Relu activation function layers, and maximum pooling layers to obtain a face plane feature map.
[0017] In one of the embodiments, the number of face plane feature maps and the number of face contour feature maps are equal.
[0018] In one embodiment, the multimodal fusion feature is enhanced according to the contrastive learning method to obtain the multimodal feature representation, including:
[0019] Select other facial plane images and contour images of the person to be identified as positive samples, select other facial plane images and contour images of other people as negative samples, perform feature extraction on the positive samples and the negative samples, and obtain the face plane feature map and face contour feature map of the positive samples and the face plane feature map and face contour feature map of the negative samples respectively;
[0020] The face plane feature map and the face contour feature map of the positive sample are cross-combined to obtain the multimodal fusion feature of the positive sample, and the face plane feature map and the face contour feature map of the negative sample are cross-combined to obtain the multimodal fusion feature of the negative sample;
[0021] According to the contrastive learning method, the similarity between the multimodal fusion features corresponding to the face image to be recognized and the multimodal fusion features of the positive sample is increased, and the similarity with the multimodal fusion features of the negative sample is reduced to obtain a multimodal feature representation.
[0022] In one embodiment, the loss function of the multimodal feature representation is
[0023]
[0024] Among them, S i represents the multimodal fusion features of the face image to be recognized, i represents the sample number of the face image to be recognized, S jrepresents the modal fusion feature of the positive sample, j represents the sample number of the positive sample, S k Does not include S i The total number of positive and negative samples of multimodal fusion features, k does not include S i The sample number of the total positive and negative samples, sim represents the L2 distance function, and τ is the parameter for adjusting the feature representation level.
[0025] In one embodiment, a 3D convolutional network is used to perform feature analysis on the multimodal feature representation to obtain a probability distribution corresponding to a person's name, including:
[0026] Using 3D convolutional network to analyze the multimodal feature representation, the probability distribution corresponding to the name is obtained:
[0027]
[0028] Among them, F r =ReLU(FC(f))=max(0,W T f+b), F r represents multimodal feature representation, f is the feature vector, W is the trainable parameter matrix, b is the bias term, and p j represents the probability that the currently recognized face is the jth person, is the jth feature parameter of the multimodal feature, is the i-th feature parameter of the multimodal feature.
[0029] In one embodiment, a cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, including:
[0030] The cross entropy loss function is constructed based on the probability distribution corresponding to the name and the annotated name label:
[0031]
[0032] Among them, y(x) represents the true distribution corresponding to the name, p(x) represents the probability distribution corresponding to the name, and x represents the name label.
[0033] A multimodal face recognition device based on contrastive learning, the device comprising:
[0034] A feature extraction module is used to obtain a face image to be identified; the face image includes a face plane image and a face contour image; a face recognition model is constructed; the face recognition model includes a convolutional neural network, a residual network, a 3D convolutional network, a fully connected layer and a SoftMax layer; the face plane image is subjected to feature extraction according to the convolutional neural network to obtain a face plane feature map; the face contour image is subjected to feature extraction using the residual network to obtain a face contour feature map;
[0035] A cross-combination module is used to cross-combine the face plane feature map and the face contour feature map to obtain a multi-modal fusion feature;
[0036] The contrastive learning module is used to enhance the representation of multimodal fusion features according to the contrastive learning method to obtain multimodal feature representation; the 3D convolutional network is used to perform feature analysis on the multimodal feature representation to obtain the probability distribution corresponding to the name;
[0037] The face recognition module is used to construct a cross entropy loss function according to the probability distribution corresponding to the name and the annotated name label, and use the cross entropy loss function and the loss function in contrastive learning to train the pre-constructed face recognition model to obtain a trained face recognition model; and perform face recognition on the face image to be recognized according to the trained face recognition model.
[0038] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0039] Acquire a face image to be recognized; the face image includes a face plane image and a face contour image;
[0040] Build a face recognition model; the face recognition model includes convolutional neural network, residual network, 3D convolutional network, fully connected layer and SoftMax layer;
[0041] Extract features of the face plane image according to the convolutional neural network to obtain a face plane feature map;
[0042] The residual network is used to extract features from the face contour image to obtain a face contour feature map;
[0043] The face plane feature map and the face contour feature map are cross-combined to obtain multi-modal fusion features;
[0044] The multimodal fusion features are enhanced according to the contrastive learning method to obtain the multimodal feature representation;
[0045] Use 3D convolutional network to perform feature analysis on multimodal feature representation and obtain the probability distribution corresponding to the name;
[0046] A cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, and the pre-constructed face recognition model is trained using the cross entropy loss function and the loss function in contrastive learning to obtain a trained face recognition model;
[0047] Perform face recognition on the face image to be identified according to the trained face recognition model.
[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0049] Acquire a face image to be recognized; the face image includes a face plane image and a face contour image;
[0050] Build a face recognition model; the face recognition model includes convolutional neural network, residual network, 3D convolutional network, fully connected layer and SoftMax layer;
[0051] Extract features of the face plane image according to the convolutional neural network to obtain a face plane feature map;
[0052] The residual network is used to extract features from the face contour image to obtain a face contour feature map;
[0053] The face plane feature map and the face contour feature map are cross-combined to obtain multi-modal fusion features;
[0054] The multimodal fusion features are enhanced according to the contrastive learning method to obtain the multimodal feature representation;
[0055] Use 3D convolutional network to perform feature analysis on multimodal feature representation and obtain the probability distribution corresponding to the name;
[0056] A cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, and the pre-constructed face recognition model is trained using the cross entropy loss function and the loss function in contrastive learning to obtain a trained face recognition model;
[0057] Perform face recognition on the face image to be identified according to the trained face recognition model.
[0058] The above-mentioned multimodal face recognition method, device, computer equipment and storage medium based on contrastive learning, the present invention performs face recognition by adding a depth map of the face contour shape to a plane image to form a multimodal feature, combines the two modal features by cross-combination when fusing the modal features, and further fuses the multimodal features when analyzing the multimodal features through a 3D convolutional network, and finally uses a contrastive learning method to enhance the representation of the multimodal features, thereby improving the accuracy of multimodal face recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 A schematic diagram of a flow chart of a multimodal face recognition method based on contrastive learning in one embodiment;
[0060] Figure 2 A schematic diagram of a framework of a multimodal face recognition method based on contrastive learning in one embodiment;
[0061] Figure 3 is a structural block diagram of a multimodal face recognition device based on contrastive learning in one embodiment;
[0062] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0064] In one embodiment, Figure 1 As shown, a multimodal face recognition method based on contrastive learning is provided, comprising the following steps:
[0065] Step 102, obtaining a face image to be recognized; the face image includes a face plane image and a face contour image; constructing a face recognition model; the face recognition model includes a convolutional neural network, a residual network, a 3D convolutional network, a fully connected layer and a SoftMax layer; extracting features from the face plane image according to the convolutional neural network to obtain a face plane feature map; extracting features from the face contour image using the residual network to obtain a face contour feature map.
[0066] The present invention uses a 2D convolutional neural network to extract features, and uses a residual network to extract features for a depth map representing the shape of a face contour. The residual network can prevent the performance of a deep model from declining when there are too many network layers. The residual module adds the input and output by adding a connection between the input and output of the network layer, which is called direct mapping. The purpose is to change the training target of the network layer from output to the difference between output and input. In this way, if the current network layer causes the model performance to deteriorate, the network layer will learn to be 0 during model training, so that the output of the network layer is equal to the input, that is, the network layer is invalidated. The present invention uses an 18-layer network structure ResNet18 composed of residual modules to extract features from a depth map representing the shape of a face contour, and an accurate face contour feature map can be obtained. The accurate extraction of face features is achieved by extracting face plane features and face contour features, which is conducive to improving the accuracy of subsequent model training, thereby improving the accuracy of face recognition.
[0067] Step 104, cross-combining the face plane feature map and the face contour feature map to obtain a multi-modal fusion feature.
[0068] The face planar feature map and the face contour feature map are cross-combined according to the positions of the corresponding channels to make the features of the two modalities more closely integrated. At the same time, in the subsequent multimodal feature analysis stage, the 3D convolution can perform convolution operations on the features of the two modalities at the same time, thereby further integrating the features of the two modalities.
[0069] Before fusion, there are multiple multimodal features. If the contrastive learning method is used to enhance the model, it will be too redundant. However, if multiple features are fused into one feature and then enhanced by contrastive learning, it will be simpler, with fewer training parameters and faster training speed. The role of fusion is to make the multimodal features have both the plane and contour features of the face during analysis, making the final recognition result more accurate and the model more secure.
[0070] Step 106, enhancing the representation of the multimodal fusion features according to the contrastive learning method to obtain a multimodal feature representation; performing feature analysis on the multimodal feature representation using a 3D convolutional network to obtain a probability distribution corresponding to the name.
[0071] Contrastive learning selects other facial plane images and depth maps of the current person to be identified as positive samples, and selects other people's facial plane images and depth maps as negative samples. The positive samples and negative samples are respectively input into the feature extraction model that is exactly the same as that in step 102, and the obtained feature maps are respectively subjected to the multimodal fusion method of step 104, so that the multimodal fusion features of the positive and negative examples are obtained respectively. Then, by comparison, the similarity between the multimodal fusion features obtained in step 104 and the positive example multimodal fusion features is shortened, and the similarity with the multimodal fusion features of the negative example is alienated, so that the multimodal fusion features are more in line with the feature representation of the current person to be identified, and the feature representation of different people can be better distinguished, thereby achieving the effect of enhancing the feature representation.
[0072] The multimodal feature representation enhanced by contrastive learning is input into a decoder composed of 3D CNN, 3D Max-pooling, activation layer, fully connected layer, and SoftMax for feature analysis. The 3D convolution uses a 3D convolution kernel to perform convolution operations with multiple feature maps on multimodal features at the same time. The 3D convolution can better model the relationship between feature maps, and can realize feature analysis while taking into account the relationship between feature maps, which is conducive to improving the accuracy of face recognition.
[0073] Step 108, constructing a cross entropy loss function based on the probability distribution corresponding to the name output by the model and the annotated real name label; using the cross entropy loss function and the loss function in contrastive learning to train the pre-constructed face recognition model to obtain a trained face recognition model; and performing face recognition on the face image to be recognized according to the trained face recognition model.
[0074] The one-hot encoding of the probability distribution of the name and the annotated name label is used to construct the cross entropy loss function. This way, the loss function can be gradually reduced by continuously adjusting the trainable parameters in the model, making the output of the model gradually close to the real category conveniently and automatically. The loss function in contrastive learning is combined with the cross entropy loss function, and the parameters are continuously updated through stochastic gradient descent to minimize the loss function. The expression is as follows: loss θ =loss cons +loss crE , Among them, loss crE represents the cross entropy loss function, θ represents all trainable parameters in the model, and η is the learning rate during model training. Model training is completed when the loss function is small enough or the set number of iterations is reached. When the model training is completed, the multimodal feature indicates that the task of the learning phase has been completed. It is necessary to replace the feature extraction model of the positive and negative samples used in contrastive learning with the loss function loss in contrastive learning. cons After the removal, the face image to be identified is subjected to face recognition. The process of performing face recognition on the face image to be identified according to the trained face recognition model is as follows: Figure 2 shown.
[0075] In the above-mentioned multimodal face recognition method based on contrastive learning, the present invention performs face recognition by adding a depth map of the face contour shape to a plane image to form a multimodal feature, combines the two modal features by cross-combination when fusing the modal features, and further fuses the multimodal features when analyzing the multimodal features through a 3D convolutional network. Finally, the contrastive learning method is used to enhance the representation of the multimodal features, thereby improving the accuracy of multimodal face recognition.
[0076] In one embodiment, feature extraction is performed on a face plane image according to a convolutional neural network to obtain a face plane feature map, including:
[0077] A 2D convolutional neural network is used to set a parameter matrix that is less than or equal to the size of the face plane image as the convolution kernel. The face plane image is feature extracted through multiple layers of convolution layers, Relu activation function layers, and maximum pooling layers to obtain a face plane feature map.
[0078] The 2D convolutional neural network first sets a parameter matrix less than or equal to the image size, called a convolution kernel. The convolution kernel slides forward and backward and up and down on the image in turn. Each time it slides, it performs a dot product with the pixel value of the image at the corresponding position, that is, feature extraction. The parameters of each convolution kernel on the same channel remain unchanged, that is, the weights are shared, and the weights of the convolution kernels on different channels are not shared. For multi-channel data, the values of each convolution kernel after the dot product at the same position of each channel are added as a pixel value of the extracted feature map, and all pixel values are gradually slid to obtain the extracted feature map. The role of the pooling operation is to reduce the dimension of the feature map. By setting a matrix frame without parameters and sliding on the feature map, all feature map pixel values in the matrix frame are converted into one value, wherein the maximum pooling is to find the maximum value as the output, and the average pooling is to calculate the average value of all values as the output. The present invention extracts the features of the face plane image through a multi-layer convolution layer, a Relu activation function layer and a maximum pooling layer to obtain a face plane feature map.
[0079] In one of the embodiments, the number of face plane feature maps and the number of face contour feature maps are equal.
[0080] In a specific embodiment, the number of face plane feature maps and face contour feature maps remains equal, paving the way for subsequent feature fusion, resulting in a better feature fusion effect.
[0081] In one embodiment, the multimodal fusion feature is enhanced according to the contrastive learning method to obtain the multimodal feature representation, including:
[0082] Select other facial plane images and contour images of the person to be identified as positive samples, select other facial plane images and contour images of other people as negative samples, perform feature extraction on the positive samples and the negative samples, and obtain the face plane feature map and face contour feature map of the positive samples and the face plane feature map and face contour feature map of the negative samples respectively;
[0083] The face plane feature map and the face contour feature map of the positive sample are cross-combined to obtain the multimodal fusion feature of the positive sample, and the face plane feature map and the face contour feature map of the negative sample are cross-combined to obtain the multimodal fusion feature of the negative sample;
[0084] According to the contrastive learning method, the similarity between the multimodal fusion features corresponding to the face image to be recognized and the multimodal fusion features of the positive sample is increased, and the similarity with the multimodal fusion features of the negative sample is reduced to obtain a multimodal feature representation.
[0085] In one embodiment, the loss function of the multimodal feature representation is
[0086]
[0087] Among them, S i represents the multimodal fusion features of the face image to be recognized, i represents the sample number of the face image to be recognized, S j represents the modal fusion feature of the positive sample, j represents the sample number of the positive sample, S k Does not include S i The total number of positive and negative samples of multimodal fusion features, k does not include S i The sample number of the total positive and negative samples, sim represents the L2 distance function, and τ is the parameter for adjusting the feature representation level.
[0088] By comparison, the similarity between the multimodal fusion features corresponding to the face image to be identified and the multimodal fusion features of the positive example is shortened, and the similarity between the multimodal fusion features of the negative example is alienated. In this way, the multimodal fusion features are more consistent with the feature representation of the current person to be identified, while being able to better distinguish the feature representations of different people, thereby achieving the effect of enhanced feature representation. In order to make the multimodal fusion features more consistent with the feature representation of the current person to be identified, while being able to better distinguish the feature representations of different people, thereby achieving the effect of enhanced feature representation, it is necessary to establish a loss function that can gradually achieve the similarity between the positive example and the negative example through continuous training.
[0089] In one embodiment, a 3D convolutional network is used to perform feature analysis on the multimodal feature representation to obtain a probability distribution corresponding to a person's name, including:
[0090] Using 3D convolutional network to analyze the multimodal feature representation, the probability distribution corresponding to the name is obtained:
[0091]
[0092] Among them, F r =ReLU(FC(f))=max(0,W T f+b), F r represents multimodal feature representation, f is the feature vector, W is the trainable parameter matrix, b is the bias term, and p j represents the probability that the currently recognized face is the jth person, is the jth feature parameter of the multimodal feature, is the i-th feature parameter of the multimodal feature.
[0093] In a specific embodiment, the multimodal fusion features enhanced by contrastive learning are input into a decoder composed of 3D CNN, 3DMax-pooling, activation layer, fully connected layer, and SoftMax for feature analysis. Although the 2D convolution operation can also be performed by convolution operation on each feature map in the multimodal feature through the 2D convolution kernel and then summed to obtain the result, the 2D convolution ignores the connection between the feature maps in the video. The present invention adopts the 3D convolution method to analyze the multimodal features. The 3D convolution uses a 3D convolution kernel to perform convolution operations with multiple feature maps on the multimodal features at the same time. Compared with the 2D convolution on each 2D channel, the 3D convolution can better model the connection between the feature maps, and can realize the feature analysis while taking into account the connection between the feature maps. The 3D pooling operation reduces the dimension of the feature map of the multimodal fusion feature and also reduces the number of feature maps. The present invention uses a multi-layer 3D CNN and a 3D pooling layer combination module to extract multiple 1*1 feature maps, i.e., feature vectors, from the multimodal fusion features. Then the feature vector is input into the fully connected layer and the Relu activation function layer. The expression is as follows:
[0094] F r =ReLU(FC(f))=max(0,W T f+b)
[0095] Among them, f is the feature vector, W is the trainable parameter matrix, and b is the bias term. Finally, the probability distribution corresponding to the name is output through the softmax function, and the name corresponding to the maximum probability is selected as the name of the face to be recognized. The expression is as follows:
[0096]
[0097] In one embodiment, a cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, including:
[0098] The cross entropy loss function is constructed based on the probability distribution corresponding to the name and the annotated name label:
[0099]
[0100] Among them, y(x) represents the true distribution corresponding to the name, p(x) represents the probability distribution corresponding to the name, and x represents the name label.
[0101] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0102] In one embodiment, Figure 3 As shown, a multimodal face recognition device based on contrastive learning is provided, comprising: a feature extraction module 302, a cross-combination module 304, a contrastive learning module 306 and a face recognition module 308, wherein:
[0103] The feature extraction module 302 is used to obtain a face image to be recognized; the face image includes a face plane image and a face contour image; a face recognition model is constructed; the face recognition model includes a convolutional neural network, a residual network, a 3D convolutional network, a fully connected layer and a SoftMax layer; the face plane image is subjected to feature extraction according to the convolutional neural network to obtain a face plane feature map; the face contour image is subjected to feature extraction using the residual network to obtain a face contour feature map;
[0104] A cross-combination module 304 is used to cross-combine the face plane feature map and the face contour feature map to obtain a multi-modal fusion feature;
[0105] A contrastive learning module 306 is used to enhance the representation of the multimodal fusion features according to the contrastive learning method to obtain a multimodal feature representation; perform feature analysis on the multimodal feature representation using a 3D convolutional network to obtain a probability distribution corresponding to the name;
[0106] The face recognition module 308 is used to construct a cross-entropy loss function according to the probability distribution corresponding to the name and the annotated name label, and use the cross-entropy loss function and the loss function in contrastive learning to train the pre-constructed face recognition model to obtain a trained face recognition model; and perform face recognition on the face image to be recognized according to the trained face recognition model.
[0107] In one embodiment, the feature extraction module 302 is further used to extract features from the face plane image according to the convolutional neural network to obtain a face plane feature map, including:
[0108] A 2D convolutional neural network is used to set a parameter matrix that is less than or equal to the size of the face plane image as the convolution kernel. The face plane image is feature extracted through multiple layers of convolution layers, Relu activation function layers, and maximum pooling layers to obtain a face plane feature map.
[0109] In one of the embodiments, the number of face plane feature maps and the number of face contour feature maps are equal.
[0110] In one embodiment, the contrastive learning module 306 is further used to enhance the representation of the multimodal fusion features according to the contrastive learning method to obtain the multimodal feature representation, including:
[0111] Select other facial plane images and contour images of the person to be identified as positive samples, select other facial plane images and contour images of other people as negative samples, perform feature extraction on the positive samples and the negative samples, and obtain the face plane feature map and face contour feature map of the positive samples and the face plane feature map and face contour feature map of the negative samples respectively;
[0112] The face plane feature map and the face contour feature map of the positive sample are cross-combined to obtain the multimodal fusion feature of the positive sample, and the face plane feature map and the face contour feature map of the negative sample are cross-combined to obtain the multimodal fusion feature of the negative sample;
[0113] According to the contrastive learning method, the similarity between the multimodal fusion features corresponding to the face image to be recognized and the multimodal fusion features of the positive sample is increased, and the similarity with the multimodal fusion features of the negative sample is reduced to obtain a multimodal feature representation.
[0114] In one embodiment, the loss function of the multimodal feature representation is
[0115]
[0116] Among them, S i represents the multimodal fusion features of the face image to be recognized, i represents the sample number of the face image to be recognized, S j represents the modal fusion feature of the positive sample, j represents the sample number of the positive sample, S k Does not include S i The total number of positive and negative samples of multimodal fusion features, k does not include S i The sample number of the total positive and negative samples, sim represents the L2 distance function, and τ is the parameter for adjusting the feature representation level.
[0117] In one embodiment, the contrastive learning module 306 is further used to perform feature analysis on the multimodal feature representation using a 3D convolutional network to obtain a probability distribution corresponding to the name, including:
[0118] Using 3D convolutional network to analyze the multimodal feature representation, the probability distribution corresponding to the name is obtained:
[0119]
[0120] Among them, F r =ReLU(FC(f))=max(0,WT f+b), F r represents multimodal feature representation, f is the feature vector, W is the trainable parameter matrix, b is the bias term, and p j represents the probability that the currently recognized face is the jth person, is the jth feature parameter of the multimodal feature, is the i-th feature parameter of the multimodal feature.
[0121] In one embodiment, the face recognition module 308 is further used to construct a cross entropy loss function according to the probability distribution corresponding to the name and the annotated name label, including:
[0122] The cross entropy loss function is constructed based on the probability distribution corresponding to the name and the annotated name label:
[0123]
[0124] Among them, y(x) represents the true distribution corresponding to the name, p(x) represents the probability distribution corresponding to the name, and x represents the name label.
[0125] For the specific definition of a multimodal face recognition device based on contrastive learning, please refer to the definition of a multimodal face recognition method based on contrastive learning above, which will not be repeated here. Each module in the above-mentioned multimodal face recognition device based on contrastive learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal face recognition method based on contrast learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0127] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0129] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0132] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A multimodal face recognition method based on contrastive learning, characterized in that: The method comprises: Acquire a face image to be identified; the face image includes a face plane image and a face contour image; Constructing a face recognition model; the face recognition model includes a convolutional neural network, a residual network, a 3D convolutional network, a fully connected layer and a SoftMax layer; Extracting features from the face plane image according to a convolutional neural network to obtain a face plane feature map; Using a residual network to extract features from the face contour image to obtain a face contour feature map; Cross-combining the face plane feature map and the face contour feature map to obtain a multi-modal fusion feature; Performing enhanced representation on the multimodal fusion features according to a contrastive learning method to obtain a multimodal feature representation; Performing feature analysis on the multimodal feature representation using a 3D convolutional network to obtain a probability distribution corresponding to the name; constructing a cross entropy loss function according to the probability distribution corresponding to the name and the annotated name label, and using the cross entropy loss function and the loss function in contrastive learning to train a pre-constructed face recognition model to obtain a trained face recognition model; Performing face recognition on the face image to be recognized according to the trained face recognition model; The multimodal fusion feature is enhanced according to a contrastive learning method to obtain a multimodal feature representation, including: Select other facial plane images and contour images of the person to be identified as positive samples, select other facial plane images and contour images of other people as negative samples, perform feature extraction on the positive samples and negative samples, and obtain face plane feature maps and face contour feature maps of the positive samples and face plane feature maps and face contour feature maps of the negative samples respectively; Cross-combining the face plane feature map and the face contour feature map of the positive sample to obtain the multimodal fusion feature of the positive sample, and cross-combining the face plane feature map and the face contour feature map of the negative sample to obtain the multimodal fusion feature of the negative sample; According to the contrastive learning method, the similarity between the multimodal fusion feature corresponding to the face image to be recognized and the multimodal fusion feature of the positive sample is increased, and the similarity with the multimodal fusion feature of the negative sample is reduced to obtain a multimodal feature representation; The multimodal feature representation is analyzed using a 3D convolutional network to obtain a probability distribution corresponding to the name, including: The 3D convolutional network is used to perform feature analysis on the multimodal feature representation, and the probability distribution corresponding to the name is obtained as follows: Among them, F r =ReLU(FC(f))=max(0,W T f+b), F r represents multimodal feature representation, f is the feature vector, W is the trainable parameter matrix, b is the bias term, and p j represents the probability that the currently recognized face is the jth person, is the jth feature parameter of the multimodal feature, is the i-th feature parameter of the multimodal feature.
2. The method according to claim 1, characterized in that The feature extraction of the face plane image is performed according to the convolutional neural network to obtain a face plane feature map, including: A 2D convolutional neural network is used to set a parameter matrix that is less than or equal to the size of the face plane image as the convolution kernel. The face plane image is subjected to feature extraction through multiple layers of convolution layers, Relu activation function layers, and maximum pooling layers to obtain a face plane feature map.
3. The method according to claim 1, characterized in that The number of the face plane feature maps and the number of the face contour feature maps are equal.
4. The method according to claim 1, characterized in that: The loss function of the multimodal feature representation is Among them, S i represents the multimodal fusion features of the face image to be recognized, i represents the sample number of the face image to be recognized, S j represents the modal fusion feature of the positive sample, j represents the sample number of the positive sample, S k Does not include S i The total number of positive and negative samples of multimodal fusion features, k does not include S i The sample number of the total positive and negative samples, sim represents the L2 distance function, and τ is the parameter for adjusting the feature representation level.
5. The method according to claim 1, characterized in that A cross entropy loss function is constructed according to the probability distribution corresponding to the name and the annotated name label, including: The cross entropy loss function is constructed based on the probability distribution corresponding to the name and the annotated name label: Among them, y(x) represents the true distribution corresponding to the name, p(x) represents the probability distribution corresponding to the name, and x represents the name label.
6. A multimodal face recognition device based on contrastive learning, characterized in that: The device comprises: A feature extraction module is used to obtain a face image to be identified; the face image includes a face plane image and a face contour image; a face recognition model is constructed; the face recognition model includes a convolutional neural network, a residual network, a 3D convolutional network, a fully connected layer and a SoftMax layer; the face plane image is subjected to feature extraction according to the convolutional neural network to obtain a face plane feature map; the face contour image is subjected to feature extraction using the residual network to obtain a face contour feature map; A cross-combination module, used for cross-combining the face plane feature map and the face contour feature map to obtain a multi-modal fusion feature; A contrastive learning module, used for enhancing the representation of the multimodal fusion features according to a contrastive learning method to obtain a multimodal feature representation, comprising: Select other facial plane images and contour images of the person to be identified as positive samples, select other facial plane images and contour images of other people as negative samples, perform feature extraction on the positive samples and negative samples, and obtain face plane feature maps and face contour feature maps of the positive samples and face plane feature maps and face contour feature maps of the negative samples respectively; Cross-combining the face plane feature map and the face contour feature map of the positive sample to obtain the multimodal fusion feature of the positive sample, and cross-combining the face plane feature map and the face contour feature map of the negative sample to obtain the multimodal fusion feature of the negative sample; According to the contrastive learning method, the similarity between the multimodal fusion features corresponding to the face image to be recognized and the multimodal fusion features of the positive sample is increased, and the similarity with the multimodal fusion features of the negative sample is reduced to obtain a multimodal feature representation; the multimodal feature representation is feature analyzed by using a 3D convolutional network to obtain a probability distribution corresponding to the name, including: The 3D convolutional network is used to perform feature analysis on the multimodal feature representation, and the probability distribution corresponding to the name is obtained as follows: Among them, F r =ReLU(FC(f))=max(0,W T f+b), F r represents multimodal feature representation, f is the feature vector, W is the trainable parameter matrix, b is the bias term, and p j represents the probability that the currently recognized face is the jth person, is the jth feature parameter of the multimodal feature, is the i-th feature parameter of the multimodal feature; The face recognition module is used to construct a cross entropy loss function according to the probability distribution corresponding to the name and the annotated name label, and use the cross entropy loss function and the loss function in contrastive learning to train a pre-constructed face recognition model to obtain a trained face recognition model; and perform face recognition on the face image to be recognized according to the trained face recognition model.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
