Face recognition model training method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a face recognition model training method and device, equipment and a storage medium, the method comprising: obtaining multiple batches of sample images, each batch of sample images including multiple occluded face images; dividing the occluded face images to obtain multiple image blocks, the multiple image blocks including image blocks corresponding to occluded regions and image blocks corresponding to non-occluded regions; performing feature extraction on each image block to obtain first image features of each image block, and performing relationship extraction on the first image features through a self-attention mechanism to obtain target image features of the occluded face images, wherein the attention values between the image blocks corresponding to the occluded regions and the image blocks corresponding to the non-occluded regions are 0; training an initial face recognition model based on the target image features to obtain a face recognition model, the face recognition model being used for recognizing target occluded face images. The present application can improve the accuracy of occluded face recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a face recognition model training method and device, equipment and a storage medium. BACKGROUND

[0002] Face recognition technology refers to the use of computer technology for analysis and comparison to recognize faces, which has been widely used in security and attendance and other application scenarios. In some special scenarios, users may need to wear masks, glasses or scarves, etc. However, wearing masks, glasses or scarves and other items will cause serious occlusion of the face, which can easily lead to inaccurate face recognition results.

[0003] To solve this problem, the prior art usually uses occluded face images and non-occluded face images to train and extract features of a face recognition model, so as to achieve recognition of occluded faces.

[0004] However, the recognition accuracy of the above-mentioned method for occluded faces is still low, and it is not possible to achieve higher-precision face recognition. SUMMARY

[0005] The present application provides a face recognition model training method, device, equipment and storage medium to solve the problem of low accuracy in recognizing occluded faces in the prior art, and to achieve more accurate recognition of occluded faces.

[0006] The present application provides a face recognition model training method, comprising:

[0007] Obtain multiple batches of sample images, each batch of sample images including multiple occluded face images;

[0008] Divide each occluded face image to obtain multiple image blocks, the multiple image blocks including an image block corresponding to an occluded area and an image block corresponding to a non-occluded area;

[0009] Extract features from each image block to obtain first image features of each image block;

[0010] Based on the first image features of each image block, extract relationships through a self-attention mechanism to obtain target image features of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is 0;

[0011] Based on the target image features, train an initial face recognition model to obtain a face recognition model, which is used to recognize target occluded face images.

[0012] The application provides a face recognition model training method, which comprises the following steps:

[0013] The target image features corresponding to each batch of sample images are input into a shared classification network in the initial face recognition model to determine first loss information;

[0014] Intra-class sample image relationship learning is performed based on the target image features corresponding to each batch of sample images to obtain class aggregation features, wherein the occluded face images in the same intra-class sample image correspond to the same user;

[0015] Second loss information is determined based on the class aggregation features;

[0016] The initial face recognition model is trained based on the first loss information and the second loss information to obtain the face recognition model.

[0017] The application provides a face recognition model training method, which comprises the following steps:

[0018] Each batch of sample images is obtained from an original sample image set by random sampling and intra-class sampling.

[0019] The application provides a face recognition model training method, which comprises the following steps:

[0020] The class aggregation features are determined by batch self-attention mechanisms corresponding to the batch of sample images based on the target image features corresponding to each batch of sample images, wherein the attention values between non-intra-class sample images are 0.

[0021] The application provides a face recognition model training method, which comprises the following steps:

[0022] The updated image features of each image block are obtained by performing relationship extraction on the first image features of each image block through a self-attention mechanism.

[0023] The updated image features of each image block are combined to obtain the target image features.

[0024] The application further provides a face recognition method, which comprises the following steps:

[0025] An occluded face image to be recognized is obtained;

[0026] The occluded face image is preprocessed to obtain a target occluded face image.

[0027] The target occluded face image is input into a face recognition model to obtain a recognition result of the occluded face image, the recognition result including successful recognition or failed recognition, and the face recognition model being trained by the training method of any one of the face recognition models.

[0028] The application further provides a face recognition model training device, comprising:

[0029] An acquisition module is configured to acquire multiple batches of sample images, each batch of sample images including multiple occluded face images.

[0030] A division module is configured to divide each occluded face image to obtain multiple image blocks, the multiple image blocks including an image block corresponding to an occluded area and an image block corresponding to a non-occluded area.

[0031] An extraction module is configured to extract features of each image block to obtain first image features of each image block.

[0032] A relationship extraction module is configured to extract relationships based on the first image features of each image block by a self-attention mechanism to obtain target image features of the occluded face image, wherein an attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is 0.

[0033] A training module is configured to train an initial face recognition model based on the target image features to obtain a face recognition model, the face recognition model being used for recognizing a target occluded face image.

[0034] The application further provides a face recognition device, comprising:

[0035] An acquisition module is configured to acquire an occluded face image to be recognized.

[0036] A preprocessing module is configured to preprocess the occluded face image to obtain a target occluded face image.

[0037] An identification module is configured to input the target occluded face image into a face recognition model to obtain a recognition result of the occluded face image, the recognition result including successful recognition or failed recognition, and the face recognition model being trained by the training method of any one of the face recognition models.

[0038] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method of the face recognition model according to any one of the above embodiments and the face recognition method according to any one of the above embodiments when executing the program.

[0039] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the training method of the face recognition model according to any one of the above embodiments and the face recognition method according to any one of the above embodiments.

[0040] The present application also provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the training method of the face recognition model according to any one of the above embodiments and the face recognition method according to any one of the above embodiments.

[0041] The training method of the face recognition model, the device, the equipment and the storage medium provided by the present application firstly acquire multiple batches of sample images, and each batch of sample images comprises multiple occluded face sample images; secondly, by dividing each occluded face image, multiple image blocks are obtained, and feature extraction is performed on each image block to obtain the first image feature of each image block; then, the self-attention mechanism is used to extract the relationship of the first image feature to obtain the target image feature of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is set to 0, so that the influence of the occluded area on the feature extraction of the non-occluded area is avoided, the face recognition model focuses more on the non-occluded area of the face, the learning of the model on the non-occluded area of the face is strengthened, the target image feature corresponding to the occluded face image obtained is more accurate, and then the face recognition model obtained by training the initial face recognition model based on the target image feature is also more accurate, thereby improving the accuracy of face recognition. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flowchart of the training method of the face recognition model provided by the embodiment of the present application;

[0044] Figure 2 is a flowchart of image preprocessing provided by the embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a training process of a face recognition model provided by an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a face recognition method provided by an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a training device of a face recognition model provided by an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a face recognition device provided by an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] In some special scenarios, the user may wear a mask, glasses or a scarf and the like to block the face, such as in winter, many users will wear a mask to go out. In this scenario, the face recognition result will be seriously affected when face recognition is performed, for example, when taking the subway by face recognition, the face of the person wearing a mask will be difficult to recognize, which brings great challenges to the application of face recognition technology.

[0052] In view of the above problems, an embodiment of the present application provides a face recognition model training method. In the method, when the face recognition model is trained, the occluded face image can be divided to obtain a plurality of image blocks, and then the first image feature of each image block is extracted. In the process of determining the target image feature of the occluded face image based on the first image feature of each image block through the self-attention mechanism, the attention value between the face occlusion area and the face non-occlusion area in the self-attention mechanism can be set to 0. In this way, the influence of the occlusion area on the feature extraction of the non-occlusion area can be avoided, so that the face recognition model focuses more on the face non-occlusion area, the learning of the model on the face non-occlusion area is strengthened, the face recognition model has stronger robustness, and the accuracy of face recognition is improved.

[0053] The following will be described in combination with Figures 1-3The method provided by the embodiment of the present application is used for training the face recognition model, and can be applied to the face recognition scene, especially the scene of recognizing the occluded face. The subject performing the method can be any terminal device in communication connection with the camera, such as a mobile phone, a computer, a computer or any other face recognition device.

[0054] Figure 1 The flowchart of the method for training the face recognition model provided by the embodiment of the present application is shown in FIG. 1. Figure 1 The method comprises the following steps.

[0055] Step 101: Obtain a plurality of batches of sample images, each batch of sample images comprising a plurality of occluded face images.

[0056] The occluded face image comprises an occluded area and a non-occluded area. The occlusion object can be a mask or a scarf, and no specific requirement is made herein. It should be noted that each batch of sample images comprises a plurality of occluded images of the same face and a plurality of occluded images of different faces. In addition, in order to improve the accuracy of the face recognition model, unoccluded face images are also included in each batch of sample images.

[0057] For example, the occlusion object is a mask. The virtual mask generation technology can be used to virtually generate the occluded face image of the original face image with the mask. For other occlusion objects, the generation method of the occluded face image is similar to the above, and will not be described herein.

[0058] In order to facilitate the subsequent training of the face recognition model and the application in the actual scene, the sample image usually needs to be preprocessed after being obtained. Specifically, Figure 2 The flowchart of the image preprocessing provided by the embodiment of the present application is shown in FIG. 2. Figure 2 The preprocessing process comprises the following steps.

[0059] Step 201: Perform face detection on the input sample image.

[0060] The face detection on the input sample image can be performed by using the network model detection method to screen out the sample image not comprising a face.

[0061] Step 202: Perform face key point detection on the sample image comprising a face.

[0062] The face key points at least comprise a left eye, a right eye, a nose tip, a left corner of the mouth and a right corner of the mouth. The detection method of the face key points is not specifically limited herein.

[0063] Step 203: Based on the face key points, perform image alignment and cropping operation on the sample image comprising a face to obtain a sample image of a fixed size and an aligned face.

[0064] Specifically, the face can be aligned by at least one of rotation, scaling or translation operation according to the detected face key points, and the aligned face image can be cropped to obtain a face image of fixed size and aligned face, for example, an image of 224*224 size can be obtained. It should be understood that the order of the cropping and alignment operations is not specifically limited here.

[0065] After obtaining the plurality of sample images in the above manner, the following step 102 is performed.

[0066] Step 102: For each occluded face image, the occluded face image is divided to obtain a plurality of image blocks, including an image block corresponding to an occluded region and an image block corresponding to a non-occluded region.

[0067] The plurality of image blocks are of the same size, and when the occluded face image is divided, it can be divided into 4*4 image blocks or 16*16 image blocks, and the size of the image block is not specifically limited.

[0068] Specifically, Figure 3 The training process of the face recognition model provided by the embodiment of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, after obtaining a plurality of occluded face images by sampling, the occluded face images can be divided according to the size of the image block to obtain a plurality of image blocks.

[0069] Since only part of the occluded face image is occluded by the occlusion, the plurality of image blocks obtained by dividing the occluded face image include an image block corresponding to an occluded region and an image block corresponding to a non-occluded region. It should be understood that since the sample image is a fixed size image obtained by cropping, and the face is usually in a fixed region of the image, when the user wears a mask or a scarf, the occlusion is usually in a fixed region of the image, so the image block corresponding to the occluded region is an image block in the fixed region. For example, an image block in the lower region of the image can be considered as an image block corresponding to an occluded region.

[0070] Further, in order to facilitate the training of the face recognition model and subsequent processing, after dividing the occluded face image, the image block corresponding to the occluded region and the image block corresponding to the non-occluded region can be marked, for example, the image block marked as 1 represents the image block corresponding to the occluded region, and the image block marked as 0 represents the image block corresponding to the non-occluded region.

[0071] Step 103: Feature extraction is performed on each image block to obtain a first image feature of each image block.

[0072] The first image features include image features of the occluded region and image features of the non-occluded region.

[0073] With reference to the foregoing Figure 3 After obtaining the plurality of image blocks, the first image features of each image block can be extracted by using the shared convolutional neural network.

[0074] At step 104, based on the first image features of each image block, relationship extraction is performed by using a self-attention mechanism to obtain target image features of the occluded face image, wherein the attention value between the image block corresponding to the occluded region and the image block corresponding to the non-occluded region is 0.

[0075] The self-attention mechanism includes a Self-Attention operation.

[0076] With reference to the foregoing Figure 3 After obtaining the first image features of each image block, the first image features can be subjected to a "masking operation" by using the self-attention mechanism to extract the relationship of the first image features, thereby obtaining new image features corresponding to each image block. The target image features of the occluded face image can be obtained by merging the new image features corresponding to each image block. During the relationship extraction by using the self-attention mechanism, the attention value between the image block corresponding to the occluded region and the image block corresponding to the non-occluded region is set to 0, which can reduce the influence of the occluded region on the non-occluded region, so that the target image features focus on the non-occluded region, thereby improving the robustness of the face recognition model in recognizing the occlusion.

[0077] Further, the process of how to perform the "masking operation" on the first image features by using the self-attention mechanism is described as follows:

[0078] For example, assuming that the occlusion of the face image is a mask, based on the distribution of the occlusion in the face region, the lower region of the occluded face image is marked as a mask occluded region, and the other regions are marked as non-mask occluded regions. For example, for a 16*16 occluded face image, the image block size is 4*4, and 16 uniformly distributed image blocks of the same size are obtained. The four image blocks in the lower part are marked as mask occluded regions, and the image blocks in the other regions are marked as non-mask occluded regions. According to the different markings of different regions of the mask occluded face image, the self-attention matrix in the self-attention mechanism can be divided into three different matrices, i.e., the attention matrix M between the mask occluded regions, the attention matrix N between the non-mask regions, and the attention matrix C of the intersection region between the mask occluded regions and the non-mask regions.

[0079] Specifically, the attention matrix between the mask occlusion regions is a set of attentions calculated between the image blocks corresponding to the mask occlusion regions and the image blocks corresponding to the mask occlusion regions, and is denoted by a coefficient marked matrix as The attention matrix between the non-mask regions is a set of attentions calculated between the image blocks corresponding to the non-mask regions and the image blocks corresponding to the non-mask regions, and is denoted by a coefficient marked matrix as The attention matrix between the mask occlusion regions and the non-mask regions is a set of attentions calculated between the image blocks corresponding to the mask occlusion regions and the image blocks corresponding to the non-mask regions, and is denoted by a coefficient marked matrix as Wherein, the "mask operation" is to set the attention values between the image blocks corresponding to the occlusion regions and the image blocks corresponding to the non-occlusion regions to 0, and the attention values between the image blocks corresponding to the occlusion regions and the image blocks corresponding to the occlusion regions, and the attention values between the image blocks corresponding to the non-occlusion regions and the image blocks corresponding to the non-occlusion regions remain the original calculated values, and the "mask operation" is equivalent to the following formula (1):

[0080]

[0081] Wherein, A M , A N and A C respectively represent the similarity matrices between the mask occlusion regions, between the non-mask regions, and between the mask occlusion regions and the non-mask regions, and the coefficient marked matrices of the above three also satisfy Here, R hw×hw represents the size of the image block.

[0082] Further, the process of how to extract the relationship of the first image feature through the self-attention mechanism to obtain a new image feature is described as follows:

[0083] First, the extraction of the first image feature is performed on all image blocks, i.e., step 103, to obtain the first image feature of each image block, which is denoted as x∈R h×w×c ;

[0084] Then, after obtaining the first image feature of each image block, the corresponding three embedding expressions are obtained through the query (Q, query), key (K, key) and value (V, value) matrices in the self-attention mechanism, which are denoted as Q, K and V respectively. The working principle can be understood as that the self-attention mechanism combines the query and the key together through attention convergence, and applies different weights to different keys to achieve different selection tendencies of the attributes. Here, it can be understood that the weights obtained by the current key and other keys correspond to the attributes of other keys.

[0085] Finally, the first image feature is relationally extracted through a self-attention mechanism, wherein the attention value between the image block corresponding to the occlusion area and the image block corresponding to the non-occlusion area is 0.

[0086] The relational extraction is shown in the following formula (2):

[0087]

[0088] wherein K T represents the transpose of the K matrix, d k represents the column number of the Q and K matrices, i.e., the vector dimension, denotes the attention matrix between windows, wherein the window can be understood as a plurality of windows of the same size obtained by dividing all image blocks (the same image), and attention mechanism is performed between a plurality of image blocks in each window.

[0089] It can be obtained from formula (2) that the new image feature corresponding to each image block is obtained, and on this basis, the new image features corresponding to all image blocks are combined, so that the target image feature corresponding to the mask-occluded face image is obtained.

[0090] Step 105: training the initial face recognition model based on the target image feature to obtain a face recognition model, wherein the face recognition model is used for recognizing the target occluded face image.

[0091] Specifically, after the target image feature is determined, the target image feature is input into the initial face recognition model, the corresponding loss information is determined, the parameters of the initial face recognition model are adjusted based on the loss information, and the above steps are iteratively executed until the loss information is minimum or the obtained model converges, and the finally obtained model is determined as the face recognition model.

[0092] The method for training the face recognition model provided by the embodiment of the present application firstly acquires multiple batches of sample images, each batch of sample images including multiple occluded face sample images; secondly, by dividing each occluded face image, multiple image blocks are obtained, and feature extraction is performed on each image block to obtain first image features of each image block; then, relationship extraction is performed on the first image features through a self-attention mechanism to obtain target image features of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is set to 0, so that the influence of the occluded area on the feature extraction of the non-occluded area can be avoided, the face recognition model focuses more on the non-occluded area of the face, the learning of the model on the non-occluded area of the face is strengthened, the target image features corresponding to the occluded face image obtained are more accurate, and then the face recognition model obtained by training the initial face recognition model based on the target image features is also more accurate, thereby improving the accuracy of face recognition.

[0093] In a possible implementation, when the face recognition model is obtained by training the initial face recognition model based on the target image features, the training can be performed in the following manner: the target image features corresponding to each batch of sample images are input into a shared classification network in the initial face recognition model to determine first loss information; relationship learning of intra-class sample images is performed based on the target image features corresponding to each batch of sample images to obtain class aggregation features, wherein the occluded face sample images in the same intra-class sample images correspond to the same user; second loss information is determined based on the class aggregation features; and the initial face recognition model is trained based on the first loss information and the second loss information to obtain the face recognition model.

[0094] The shared classification network is a fully connected neural network classifier, and the first loss information and the second loss information at least include the size of a loss value.

[0095] Continuing to refer to Figure 3 The target image features obtained through step 104 are input into the shared classification network, and the first loss value is calculated.

[0096] Further, the class aggregation features corresponding to the same intra-class sample images are obtained through the relationship learning of the intra-class sample images in the shared classification network. Specifically, similar to the self-attention mechanism of the previous "mask operation", the self-attention mechanism of the "class mask operation" can also be used to extract the relationship of the target image features corresponding to the sample images to realize the relationship learning of the intra-class samples. The difference is that the attention matrix here is divided according to the relationship between the intra-class samples and the non-intra-class samples. Specifically, the attention matrix between the intra-class samples and the intra-class samples is the attention set calculated between the same class samples; the attention matrix between the non-intra-class samples and the non-intra-class samples is the attention set calculated by random sampling. It should be noted that, in order to further improve the learning efficiency of the intra-class samples and obtain more aggregated class aggregation features, the attention matrix between the non-intra-class samples and the non-intra-class samples can be set to a zero matrix, and the attention matrix between the intra-class samples and the intra-class samples is kept as the original calculated value.

[0097] Further, the self-attention mechanism of the "class mask operation" is applied to the target image features corresponding to each batch of sample images to obtain the class aggregation features corresponding to the same intra-class sample images, and then the class aggregation features are also input into the shared classification network to calculate the second loss value. The first loss information and the second loss information are used as the final loss information, so that the initial face recognition model is trained based on the loss information, and thus a final face recognition model is obtained.

[0098] In the embodiment, by inputting the target image features corresponding to each batch of sample images into the shared classification network to calculate the first loss information, and inputting the class aggregation features obtained through the intra-class sample relationship learning into the shared classification network to calculate the second loss information, and on this basis, the initial face recognition model is trained based on the first loss information and the second loss information obtained by calculation, to obtain a more accurate face recognition model, thereby improving the accuracy of face recognition.

[0099] On the basis of the above embodiment, in order to obtain a more accurate face recognition model, a plurality of batches of sample images can be obtained in the following manner to ensure that there are sufficient intra-class sample quantities in the training process of the face recognition model, thereby serving as intra-class relationship learning:

[0100] Random sampling and intra-class sampling are used to obtain each batch of sample images from the original sample image set.

[0101] Specifically, the acquisition of multiple batches of sample images needs to resort to two sampling methods: one is a traditional sampling method, i.e., random sampling, which is used to improve the accuracy of model training. The other is an intra-class sampling method, which is used to ensure the number of intra-class samples to improve the effectiveness of model training. Here, random sampling and intra-class sampling need to be sampled by two independent data readers. In order to facilitate the unified acquisition of multiple batches of sample images, the sample data read by the two data readers can be merged into one batch processing unit. It should be noted that the batch processing unit is a data set for one forward propagation in the model training process. In other words, the output of random sampling is taken as the input of intra-class sampling, and the output of intra-class sampling is calculated until the output layer is calculated.

[0102] Further, by combining random sampling and intra-class sampling for sampling to acquire multiple batches of sample images, on the one hand, the accuracy of the face recognition model can be improved, and on the other hand, the number of intra-class samples can be ensured, and the effectiveness of the face recognition model training is improved, so that the face recognition result is more accurate.

[0103] For example, on the basis of any of the above embodiments, in order to further improve the accuracy of the face recognition model training, when the relationship learning of the intra-class sample images based on the respective target image features corresponding to each batch of sample images is performed to obtain the class aggregated features, the following method can be used to acquire:

[0104] Based on the respective target image features corresponding to each batch of sample images, the class aggregated features are determined by the batch self-attention mechanism corresponding to the batch of sample images, wherein the attention values between non-intra-class sample images are 0.

[0105] Wherein, the batch self-attention mechanism can be understood as a special self-attention mechanism, which can be used to extract the class aggregated features of the images in the batch processing unit. It should be noted that this self-attention mechanism cannot directly process the images in the batch processing unit, but needs to first extract the target image features corresponding to each batch of sample images in the batch processing unit by the aforementioned "mask operation" self-attention mechanism, and then extract the class aggregated features based on the target image features.

[0106] Further, the detailed process of how to determine the class aggregated features based on the respective target image features corresponding to each batch of sample images by the batch self-attention mechanism corresponding to the batch of sample images is as follows:

[0107] Specifically, as Figure 3As shown, after obtaining the target image features corresponding to each batch of sample images respectively, the batch self-attention mechanism of the "class mask operation" can be used to extract the relationship of the target image features, so as to extract the class aggregation features corresponding to the sample images in the same class. The batch self-attention mechanism can be performed by the following formula (3) and formula (4):

[0108]

[0109]

[0110] wherein, is the class aggregation feature corresponding to the sample images in the same class, Y∈R B×C , represents the target image features corresponding to each batch of sample images respectively, LN represents layer normalization, MSA represents multi-headed self-attention mechanism, and MLP represents multi-layer perception.

[0111] Further, in order to improve the training efficiency of the face recognition model, the Transformer encoder with parallel processing mechanism can be used to establish the model of the sample relationship in the same class. The Transformer encoder includes multi-headed self-attention mechanism (MSA), layer normalization (LN), and multi-layer perception (MLP).

[0112] In addition, when the batch self-attention mechanism is used to extract the relationship of the target image features, the class mask operation needs to be performed on the batch attention matrix with the known sample class label, that is, the attention values between the non-class sample images in each batch of sample images are set to 0. The batch attention matrix is the attention matrix generated in the batch self-attention mechanism, which represents the correlation between samples.

[0113] In this embodiment, by setting the attention values between the non-class sample images in the batch self-attention mechanism to 0, the batch self-attention mechanism can effectively extract the class aggregation features corresponding to the sample images in the same class from the target image features, so that the feature aggregation between the sample images in the same class can be performed, the robustness of the neural network to mask occlusion is improved, and the accuracy and performance of the face recognition model are improved, so that the face recognition result is more accurate.

[0114] In a possible implementation, when the target image features of the occluded face image are obtained by performing relationship extraction on the first image features of each image block through the self-attention mechanism, the following methods can be used:

[0115] Based on the first image feature of each image block, relationship extraction is performed through a self-attention mechanism to obtain an updated image feature of each image block, and the updated image features of each image block are combined to obtain a target image feature.

[0116] Specifically, after obtaining the first image feature of each image block, the self-attention mechanism with the mask operation described in the foregoing embodiments can be used to perform relationship extraction on the first image feature to obtain a new image feature of each image block. The new image feature can be directly generated based on the original first image feature, or can be generated by replacing the original first image feature. In this way, a new image feature of each image block can be obtained, and the updating manner of the first image feature is not limited here.

[0117] Further, after the first image feature of each image block is updated, all updated image features can be combined to obtain a target image feature. Of course, in order to speed up the training of the face recognition model, the updated image features can also be combined in a manner of updating and combining at the same time to quickly obtain the target image feature. That is, the combination of the updated image features is completed according to the updating order of the first image feature, and the combination manner is not limited here.

[0118] In this embodiment, the first image feature of each image block is subjected to relationship extraction through a self-attention mechanism to obtain an updated image feature of each image block, and the updated image features are combined to form a target image feature, thereby providing a training basis for subsequent training of the initial face recognition model and providing an adjustment basis for adjustment of the initial face recognition model, thereby indirectly improving the accuracy of the face recognition model.

[0119] Figure 4 A flowchart of a face recognition method provided by the embodiment of the present application is shown in FIG. 1. Figure 4 The method comprises the following steps.

[0120] Step 401: Obtain a blocked face image to be recognized.

[0121] The blocked face image can include a mask, a scarf, or other similar blocking objects.

[0122] Specifically, the blocked face image can be obtained through network downloading or camera acquisition, and the acquisition manner of the image is not limited here.

[0123] Step 402: Preprocess the blocked face image to obtain a target blocked face image.

[0124] Specifically, after obtaining the blocked face image, the blocked face image can be preprocessed through a face detection algorithm, a face alignment algorithm, a face enhancement algorithm, or a face segmentation algorithm. Figure 2The image preprocessing method shown in the method is used to preprocess the occluded face image to obtain an occluded face image of a fixed size and an aligned face, i.e., a target occluded face image.

[0125] Step 403: inputting the target occluded face image into the face recognition model to obtain a recognition result of the occluded face image, the recognition result including a successful recognition or a failed recognition, the face recognition model being trained by the training method of the face recognition model in any of the above embodiments.

[0126] Specifically, the target occluded face image obtained in step 402 and the plurality of unoccluded face images to be compared can be input into the face recognition model to perform forward transmission by using the Transformer network in the face recognition model, and then the face image features of the target occluded face image and the plurality of unoccluded face images to be compared can be obtained by the output of the feature layer in the face recognition model. On this basis, the face image features of the target occluded face image to be recognized and the face image features of the plurality of unoccluded face images to be compared are compared by calculating the cosine distance between the face image features, and further, the cosine distance can be converted into a similarity score for comparison to select the maximum similarity and compare it with a similarity threshold. If the comparison result is greater than the threshold, the unoccluded face image corresponding to the maximum similarity is the target face, that is, the identity information recognition of the target occluded face image is successful, otherwise, the recognition is failed.

[0127] In the embodiment, the obtained occluded face image is preprocessed to obtain a target occluded face image, and the target occluded face image is input into a more accurate face recognition model to obtain a more accurate recognition result of the occluded face image, thereby improving the accuracy of face recognition. The face recognition model is trained by the training method of the face recognition model in any of the above embodiments.

[0128] The training device of the face recognition model provided in the embodiment of the application is described below. The training device of the face recognition model described below can be correspondingly referred to the training method of the face recognition model described above.

[0129] Figure 5 The structure diagram of the training device of the face recognition model provided in the embodiment of the application is shown in FIG. 5, which includes: Figure 5

[0130] The obtaining module 510 is configured to obtain a plurality of batches of sample images, each batch of sample images including a plurality of occluded face images.

[0131] ​The dividing module 520 is configured to divide each occluded face image to obtain a plurality of image blocks, wherein the plurality of image blocks include an image block corresponding to an occluded area and an image block corresponding to a non-occluded area.

[0132] The extracting module 530 is configured to perform feature extraction on each image block to obtain a first image feature of each image block.

[0133] The relationship extracting module 540 is configured to perform relationship extraction on the first image feature of each image block by using a self-attention mechanism to obtain a target image feature of the occluded face image, wherein an attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is 0.

[0134] The training module 550 is configured to train an initial face recognition model based on the target image feature to obtain a face recognition model, wherein the face recognition model is used for recognizing a target occluded face image.

[0135] The training device for the face recognition model provided by the embodiment of the present application firstly acquires a plurality of batches of sample images by using the acquiring module 510, wherein each batch of sample images includes a plurality of occluded face sample images; secondly, each occluded face image is divided by using the dividing module 520 to obtain a plurality of image blocks, and feature extraction is performed on each image block by using the extracting module 530 to obtain a first image feature of each image block; then, relationship extraction is performed on the first image feature by using the relationship extracting module 540 in the self-attention mechanism to obtain a target image feature of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is set to 0, so that the influence of the occluded area on the feature extraction of the non-occluded area can be avoided, the face recognition model focuses more on the non-occluded area of the face, the learning of the model on the non-occluded area of the face is strengthened, the target image feature corresponding to the occluded face image obtained is more accurate, and then the face recognition model obtained by training the initial face recognition model by using the training module 550 based on the target image feature is also more accurate, thereby improving the accuracy of face recognition.

[0136] Optionally, the training module 550 is specifically configured to:

[0137] input the target image feature corresponding to each batch of sample images into a shared classification network in the initial face recognition model to determine first loss information;

[0138] perform relationship learning on intra-class sample images based on the target image feature corresponding to each batch of sample images to obtain class aggregation features, wherein the occluded face sample images in the same intra-class sample images correspond to the same user.

[0139] determine second loss information based on the class aggregation feature;

[0140] train the initial face recognition model based on the first loss information and the second loss information to obtain a target face recognition model.

[0141] Optionally, the obtaining module 510 is specifically configured to:

[0142] The original sample image set is obtained in a random sampling and intra-class sampling manner.

[0143] Optionally, the training module 550 is further specifically configured to:

[0144] The class aggregation feature is determined based on the target image feature corresponding to each batch of sample images through a batch self-attention mechanism corresponding to the batch of sample images, wherein the attention value between non-intra-class sample images is 0.

[0145] Optionally, the relationship extraction module 540 is specifically configured to:

[0146] The updated image feature of each image block is obtained through self-attention mechanism based on the first image feature of each image block;

[0147] The target image feature is obtained by combining the updated image feature of each image block.

[0148] The device of the embodiment can be used to execute the method of any one of the face recognition model training device side method embodiments, and the specific implementation process and technical effects are similar to those of the face recognition model training device side method embodiments. For details, refer to the face recognition model training device side method embodiments, which will not be described here.

[0149] Further, Figure 6 A structural diagram of a face recognition device according to an embodiment of the present application is shown in FIG. 6. Figure 6 As shown in FIG. 6, the device comprises:

[0150] The obtaining module 610 is configured to obtain a to-be-identified occluded face image.

[0151] The preprocessing module 620 is configured to pre-process the occluded face image to obtain a target occluded face image.

[0152] The recognition module 630 is configured to input the target occluded face image into a face recognition model to obtain a recognition result of the occluded face image, wherein the recognition result comprises a recognition success or a recognition failure, and the face recognition model is trained based on the face recognition model training method according to any one of the above methods.

[0153] In this embodiment, the occluded face image to be identified is acquired by the acquisition module 610; then, the acquired occluded face image is preprocessed by the preprocessing module 620 to obtain a target occluded face image, and the target occluded face image is input to a more accurate face recognition model, so as to identify the target occluded face image by the identification module 630 to obtain a more accurate identification result of the occluded face image, thereby improving the accuracy of face recognition. The face recognition model is obtained by training the face recognition model according to any one of the embodiments of the face recognition model training method.

[0154] The device of this embodiment can be used to execute the method of any one of the face recognition device-side method embodiments, and the specific implementation process and technical effects are similar to those of the face recognition device-side method embodiments. For details, refer to the detailed description in the face recognition device-side method embodiments, which will not be repeated here.

[0155] Figure 7 An example of a schematic diagram of the physical structure of an electronic device is shown in FIG. 8. Figure 7 As shown in FIG. 8, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740. The processor 710, the communications interface 720, and the memory 730 can communicate with each other through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute a face recognition model training method. The method includes acquiring multiple batches of sample images, each batch of sample images including multiple occluded face images; for each occluded face image, dividing the occluded face image to obtain multiple image blocks, the multiple image blocks including an image block corresponding to an occluded area and an image block corresponding to a non-occluded area; performing feature extraction on each image block to obtain a first image feature of each image block; based on the first image feature of each image block, performing relationship extraction through a self-attention mechanism to obtain a target image feature of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is 0; based on the target image feature, training an initial face recognition model to obtain a face recognition model, the face recognition model being used for identifying a target occluded face image.

[0156] In addition, the logic instructions in the memory 730 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art that contributes essentially or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0157] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the face recognition model training method provided by the above-mentioned method, the method comprising: obtaining multiple batches of sample images, each batch of sample images comprising multiple occluded face images; for each occluded face image, dividing the occluded face image to obtain multiple image blocks, the multiple image blocks comprising an image block corresponding to an occluded area and an image block corresponding to a non-occluded area; performing feature extraction on each image block to obtain a first image feature of each image block; based on the first image feature of each image block, performing relationship extraction through a self-attention mechanism to obtain a target image feature of the occluded face image, wherein the attention value between the image block corresponding to the occluded area and the image block corresponding to the non-occluded area is 0; based on the target image feature, training an initial face recognition model to obtain a face recognition model, the face recognition model being used for recognizing a target occluded face image.

[0158] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method for training a face recognition model as described above, the method comprising: obtaining a plurality of batches of sample images, each batch of sample images comprising a plurality of occluded face images; for each occluded face image, dividing the occluded face image to obtain a plurality of image blocks, the plurality of image blocks comprising an image block corresponding to an occluded region and an image block corresponding to a non-occluded region; performing feature extraction on each image block to obtain a first image feature of each image block; performing relationship extraction on the first image feature of each image block based on a self-attention mechanism to obtain a target image feature of the occluded face image, wherein an attention value between the image block corresponding to the occluded region and the image block corresponding to the non-occluded region is 0; and training an initial face recognition model based on the target image feature to obtain the face recognition model, the face recognition model being used for recognizing a target occluded face image.

[0159] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0160] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a face recognition model, characterized in that, include: Acquire multiple batches of sample images, each batch containing multiple images of occluded faces; For each occluded face image, the occluded face image is divided into multiple image blocks, including image blocks corresponding to the occluded area and image blocks corresponding to the non-occluded area. Feature extraction is performed on each image patch to obtain the first image feature of each image patch; Based on the first image features of each image block, a relationship is extracted through a self-attention mechanism to obtain the target image features of the occluded face image, wherein the attention value between the image block corresponding to the occluded region and the image block corresponding to the non-occluded region is 0. Based on the features of the target image, an initial face recognition model is trained to obtain a face recognition model, which is used to recognize the occluded face image of the target. The step of training an initial face recognition model based on the target image features to obtain a face recognition model includes: The target image features corresponding to each batch of sample images are input into the shared classification network in the initial face recognition model to determine the first loss information; Based on the target image features corresponding to each batch of sample images, the relationship between sample images within a class is learned to obtain class aggregation features, wherein occluded face images in the same batch of sample images correspond to the same user. Based on the aforementioned class aggregation features, the second loss information is determined; Based on the first loss information and the second loss information, the initial face recognition model is trained to obtain the face recognition model.

2. The training method for the face recognition model according to claim 1, characterized in that, The acquisition of multiple batches of sample images includes: Each batch of sample images is obtained from the original sample image set using random sampling and intra-class sampling methods.

3. The training method for the face recognition model according to claim 2, characterized in that, The process of learning the relationship between sample images within a class based on the target image features corresponding to each batch of sample images to obtain class aggregation features includes: Based on the target image features corresponding to each batch of sample images, the class aggregation features are determined through the batch self-attention mechanism corresponding to the batch of sample images, wherein the attention value between non-intra-class sample images is 0.

4. The training method for the face recognition model according to any one of claims 1-3, characterized in that, The step of extracting the target image features of the occluded face image by performing relation extraction through a self-attention mechanism based on the first image features of each image block includes: Based on the first image features of each image block, relation extraction is performed through a self-attention mechanism to obtain the updated image features of each image block; The updated image features of each image block are combined to obtain the target image features.

5. A face recognition method, characterized in that, include: Obtain the image of the occluded face to be identified; The occluded face image is preprocessed to obtain the target occluded face image; The target occluded face image is input into a face recognition model to obtain the recognition result of the occluded face image. The recognition result includes successful recognition or recognition failure. The face recognition model is trained based on the face recognition model training method described in any one of claims 1-4.

6. A training device for a face recognition model, characterized in that, include: The acquisition module is used to acquire multiple batches of sample images, each batch of sample images including multiple occluded face images; The segmentation module is used to segment each occluded face image into multiple image blocks, including image blocks corresponding to the occluded area and image blocks corresponding to the non-occluded area. The extraction module is used to extract features from each image patch to obtain the first image features of each image patch; The relation extraction module is used to extract relations based on the first image features of each image block through a self-attention mechanism to obtain the target image features of the occluded face image, wherein the attention value between the image block corresponding to the occluded region and the image block corresponding to the non-occluded region is 0. A training module is used to train an initial face recognition model based on the target image features to obtain a face recognition model, which is used to recognize occluded face images. The process of training the initial face recognition model based on the target image features to obtain the face recognition model includes: inputting the target image features corresponding to each batch of sample images into a shared classification network in the initial face recognition model to determine first loss information; performing intra-class sample image relationship learning based on the target image features corresponding to each batch of sample images to obtain class aggregation features, wherein occluded face images in the same intra-class sample images correspond to the same user; determining second loss information based on the class aggregation features; and training the initial face recognition model based on the first loss information and the second loss information to obtain the face recognition model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method of the face recognition model as described in any one of claims 1 to 4, and the face recognition method as described in claim 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the face recognition model as described in any one of claims 1 to 4, and the face recognition method as described in claim 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the face recognition model as described in any one of claims 1 to 4, and the face recognition method as described in claim 5.

Images