Model training method, face recognition method, device, equipment and medium

Abstract

The application discloses a model training method, a face recognition method, a device, equipment and a medium, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring face images of a target object in N continuous frames; inputting the face images into a face recognition model for feature extraction to obtain first face feature information and second face feature information; the face extraction model comprises X layers of feature extraction layers; the first face feature information is obtained after the X layers of feature extraction layers are processed in sequence; the second face feature information is obtained after the i layer in the X layers of feature extraction layers is processed; i is an element in a set {2, 3,..., X-1}; a target loss parameter is calculated based on the first face image feature information and the second face feature information; and the face recognition model is trained based on the target loss parameter to obtain a trained face recognition model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, specifically relating to a model training method, a face recognition method, a device, equipment, and a medium. Background Technology

[0002] With the development of electronic technology, when users lend their electronic devices to others and cannot control the other users' operation of the devices, there is a significant risk of privacy leaks. Currently, electronic devices can be set to perform real-time facial recognition for specific applications. This means that the device continuously recognizes the user's face, and if the application is used without matching the preset face, the application will be suspended on the device.

[0003] In existing technologies, the training method for real-time face recognition is usually based on a face database. During training, face images obtained from the database are used to extract features from the face images through a face recognition model to obtain the face feature information corresponding to the face image. The face image corresponding to the face feature information in the database is then compared with the input face image to calculate the comparison parameters. Finally, the face recognition model is fine-tuned using these parameters.

[0004] Thus, because the comparison parameters used to adjust the face recognition model in the existing face recognition training methods are too simple and fixed, it is impossible to obtain more accurate comparison parameters. As a result, the pass rate and stability of face recognition remain low after the face recognition model is updated. Summary of the Invention

[0005] The purpose of this application is to provide a model training method, face recognition method, device, equipment, and medium that can solve the problem that the pass rate and stability of face recognition are still low after the face recognition model is updated.

[0006] In a first aspect, embodiments of this application provide a method for training a face recognition model. This method includes: acquiring N consecutive frames of face images of a target object; inputting the face images into a face recognition model for feature extraction to obtain first face feature information and second face feature information; the face extraction model includes X feature extraction layers; the first face feature information is obtained after sequential processing by the X feature extraction layers, and the second face feature information is obtained after processing by the i-th layer of the X feature extraction layers; i∈{2, 3, ..., X-1}; calculating a target loss parameter based on the first and second face feature information; and training the face recognition model based on the target loss parameter to obtain a new face recognition model.

[0007] Secondly, embodiments of this application provide a training apparatus for a face recognition model, comprising: an acquisition module and a processing module; the acquisition module is used to acquire face images of a target object in N consecutive frames; the processing module is used to input the aforementioned N consecutive frames of the target object's face images into a face recognition model for feature extraction, obtaining first face feature information and second face feature information; the face extraction model includes X feature extraction layers; the aforementioned first face feature information is obtained after sequential processing by the aforementioned X feature extraction layers, and the aforementioned second face feature information is obtained after processing by the i-th layer in the aforementioned X feature extraction layers; i∈{2, 3, ..., X-1}; the aforementioned processing module is further used to calculate a target loss parameter based on the aforementioned first face feature information and the aforementioned second face feature information; the aforementioned processing module is further used to train the face recognition model based on the aforementioned target loss parameter to obtain a new face recognition model; wherein, N and X are positive integers greater than 1.

[0008] Thirdly, embodiments of this application provide a face recognition method based on a face recognition model. This face recognition method includes: acquiring a current frame face image; detecting key point information of the current frame face image; processing the current frame face image based on the key point information and inputting it into a face recognition model, which then processes the image through an X-layer feature extraction layer to obtain third face feature information; matching the third face feature information with preset face feature information and outputting a first recognition result; wherein the face recognition model is a face recognition model trained using the training method of the face recognition model described in the first aspect.

[0009] Fourthly, embodiments of this application provide a face recognition device based on a face recognition model, which includes: an acquisition module, a detection module, and a processing module; the acquisition module is further configured to acquire a face image of the current frame; the detection module is configured to detect key point information of the face image of the current frame; the processing module is further configured to process the face image of the current frame based on the key point information and input it into the face recognition model, and process it sequentially through the X-layer feature extraction layer of the face recognition model to obtain third face feature information; the processing module is further configured to match the third face feature information with preset face feature information and output a first recognition result; wherein, the face recognition model is a face recognition model trained using the training method of the face recognition model in the first aspect.

[0010] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect or the method described in the third aspect.

[0011] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect or the method described in the third aspect.

[0012] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method described in the first aspect, or the method described in the third aspect.

[0013] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect or the method described in the third aspect.

[0014] In this embodiment, N consecutive frames of facial images of the target object are acquired; the facial images are input into a facial recognition model for feature extraction to obtain first facial feature information and second facial feature information; the facial extraction model includes X feature extraction layers; the first facial feature information is obtained after sequential processing by the X feature extraction layers, and the second facial feature information is obtained after processing by the i-th layer in the X feature extraction layers; i∈{2, 3, ..., X-1}; based on the first facial feature information and the second facial feature information, a target loss parameter is calculated; based on the target loss parameter, the facial recognition model is trained to obtain a trained facial recognition model. In this way, by inputting multiple face images, we can obtain first face feature information and second face feature information, which are used to represent face feature information at different levels. Based on the feature information of these two, we obtain the target loss parameters of the face recognition model and train the face recognition model to obtain a trained face recognition model. Since the face recognition model obtains face feature information at different levels and calculates the loss of these face feature information, and applies the loss to train the face recognition model, the face recognition model can recognize the face feature information of different faces more accurately, thereby improving the pass rate and stability of face recognition. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a training method for a face recognition model provided in an embodiment of this application;

[0016] Figure 2 This is a schematic diagram of the structure of a face recognition model provided in an embodiment of this application;

[0017] Figure 3 This is a flowchart illustrating a face recognition method based on a face recognition model provided in an embodiment of this application;

[0018] Figure 4 This is a schematic diagram illustrating an example of a face recognition method based on a face recognition model provided in an embodiment of this application;

[0019] Figure 5 This is a schematic diagram of the structure of a training device for a face recognition model provided in an embodiment of this application;

[0020] Figure 6 This is a schematic diagram of the structure of a face recognition device based on a face recognition model provided in an embodiment of this application;

[0021] Figure 7 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0022] Figure 8 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The model training method, face recognition method, device, equipment, and medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0026] In existing technologies, the training method for real-time face recognition is usually based on a face database. During training, face images obtained from the database are used to extract features from the face images through a face recognition model to obtain the face feature information corresponding to the face image. The face image corresponding to the face feature information in the database is then compared with the input face image to calculate the comparison parameters. Finally, the face recognition model is fine-tuned using these parameters.

[0027] In this embodiment, N consecutive frames of facial images of the target object, i.e., multiple facial images, are acquired. These N frames of facial images are input into a facial recognition model for feature extraction to obtain first facial feature information and second facial feature information. The facial extraction model includes X feature extraction layers. The second facial feature information is obtained after processing by the 13th layer of the feature extraction layers. The classification cross-entropy loss and the first L2 loss are calculated based on the first facial feature information, and the second L2 loss is calculated based on the second facial feature information. The target loss parameter is then calculated based on these three losses. The facial recognition model is trained according to the target loss parameter to obtain a new facial recognition model. In this way, by inputting multiple face images, we can obtain first face feature information and second face feature information, which are used to represent face feature information at different levels. Based on the feature information of these two, we obtain the target loss parameters of the face recognition model and train the face recognition model to obtain a trained face recognition model. Since the face recognition model obtains face feature information at different levels and calculates the loss of these face feature information, and applies the loss to train the face recognition model, the face recognition model can recognize the face feature information of different faces more accurately, thereby improving the pass rate and stability of face recognition.

[0028] The face recognition model training method and the execution entity of the face recognition method based on the face recognition model provided in this application embodiment can be a face recognition model training device, which can be an electronic device or a functional module in an electronic device. The following uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0029] This application provides a method for training a face recognition model. Figure 1 A flowchart illustrating a training method for a face recognition model provided in an embodiment of this application is shown. Figure 1 As shown, the training method for the face recognition model provided in this application embodiment may include the following steps 201 to 204.

[0030] Step 201: Obtain face images of the target object in N consecutive frames.

[0031] Where N is a positive integer greater than 1.

[0032] In this embodiment of the application, the face images of the target object in the above-mentioned consecutive N frames can be real-time face images obtained by an electronic device, or they can be obtained from a training dataset.

[0033] In one example, when a user uses an electronic device, the device acquires sequential facial images of the user, for example, every 200ms.

[0034] In another example, all the facial images of different users were used to construct the training dataset.

[0035] For example, the training dataset described above may include face images of different users, or it may include face images of only a single user (i.e., the target object described above).

[0036] For example, during the training of a face recognition model, different groups of face images need to be constructed. Therefore, some face images in the training samples can be randomly replaced with face images of other users. It should be noted that the newly added face images in the training samples are not from the same user as those in the training samples.

[0037] For example, during the training of a face recognition model, face images of the same user within the same consecutive time period can be treated as a sequence. From this sequence, N consecutive frames of face images are sampled as training samples for the face recognition model. It should be noted that there can be overlapping frames between training samples.

[0038] Step 202: Input the acquired N consecutive frames of face images into the face recognition model for feature extraction to obtain the first face feature information and the second face feature information.

[0039] In this embodiment, the face recognition model includes an X-layer feature extraction layer, where X is a positive integer greater than 1.

[0040] In the embodiments of this application, such as Figure 2 As shown, the face recognition model described above may include an X-layer feature extraction layer, different convolutional layers, an L2 norm layer, and a softmax layer.

[0041] In one example, the face recognition model described above includes a feature extraction module, which includes the aforementioned X-layer feature extraction layer.

[0042] In another example, the aforementioned X feature extraction layers are X bottleneck structures.

[0043] In this embodiment of the application, the first facial feature information is obtained after being processed sequentially by the X-layer feature extraction layer.

[0044] In this embodiment of the application, the second facial feature information is obtained after processing by the i-th layer in the X-layer feature extraction layer, where i∈{2,3,……14}.

[0045] For example, the i-th feature extraction layer is one of the X-layer feature extraction layers.

[0046] It should be noted that the i-th feature extraction layer mentioned above can be called the intermediate layer of the feature extraction module, and the second face feature information mentioned above can be called the intermediate layer feature extracted from the i-th layer.

[0047] Thus, since sequential facial images of the same user not only possess semantic consistency but also exhibit high consistency in details such as facial position and contour, and because high-level features in neural networks contain more semantic information, lower-level features contain more detailed information, such as position-related contour textures, it is necessary to extract intermediate feature information from multiple feature extraction layers to better capture the detailed features of facial images and achieve more accurate facial recognition.

[0048] Optionally, in this embodiment of the application, the above-mentioned step 202, "inputting the above-mentioned face image into a face recognition model for feature extraction to obtain first face feature information and second face feature information," includes steps 202a to 202c:

[0049] Step 202a: Input the above face image into the above X-layer feature extraction layer for sequential processing to obtain the first face feature image.

[0050] Step 202b: After performing a convolution transformation on the first face feature image, the first face feature information is obtained.

[0051] For example, each feature extraction layer consists of a convolutional layer (conv), a normalization layer (batch norm), and a non-linear activation layer (relu).

[0052] Specifically, the electronic device first inputs the aforementioned face image into an X-layer feature extraction layer. After passing through the convolutional layers (conv, batch normalization, and ReLU) in the X-layer feature extraction layer, a first face feature image (e.g., a face feature image of size 128x7x7) is extracted. Next, this first face feature image is sequentially input into convolutional layers of different sizes (e.g., 1x1 conv, depth-wise conv, 1x1 conv), thereby extracting face feature information from each channel of the first face feature image through these convolutional layers. Then, the face feature information from each channel is combined. Finally, L2 normalization is applied to the combined face feature information to normalize it, obtaining the first face feature information.

[0053] Step 202c: The second face feature image obtained after processing by the i-th feature extraction layer is processed by a convolutional layer to obtain the second face feature information.

[0054] For example, firstly, the electronic device extracts a second face feature image (e.g., a face feature image of size 128x7x7) from the i-th layer of the X layer of the feature extraction module. Then, the second face feature image is input into a convolutional layer (e.g., a 7x7 depth-wise conv). The convolutional layer pools the face feature information in the second face feature image, thereby removing redundant content in the face feature information, reducing the computational parameters, and thus obtaining more accurate face feature information (e.g., the face feature information can be 128x1x1 in size). Then, the face feature information from different channels is combined through another convolutional layer (e.g., a 1x1 conv). Finally, the combined face feature information is subjected to L2 norm transformation, thereby normalizing the face feature information to obtain the second face feature information.

[0055] Step 203: Based on the first facial feature information and the second facial feature information mentioned above, the target loss parameters are calculated.

[0056] Step 204: Based on the above target loss parameters, train the face recognition model to obtain a trained face recognition model.

[0057] Optionally, in this embodiment of the application, step 203, "calculating the target loss parameter based on the first facial feature information and the second facial feature information," includes steps 203a to 203c:

[0058] Step 203a: Based on the first face feature information mentioned above, obtain the classification cross-entropy loss and the first loss.

[0059] For example, the first loss mentioned above may be a first L2 loss.

[0060] For example, the aforementioned first facial feature information is transformed into probability values ​​for each category through a 128xN fully connected layer (N being the number of ID categories in the training set) and a Softmax layer. Combined with the true label of the image, the margin-based classification cross-entropy loss Lce is obtained.

[0061] For example, the L2 loss is calculated based on the first formula.

[0062] For example, the first formula above is: .

[0063] Where i is the i-th frame of the face image of the target object in N frames.

[0064] As a further example, the first L2 loss is calculated based on the second formula as the sum of the L2 losses between each pair of the first facial feature information corresponding to the N frames of the target object's face image input to the face recognition model.

[0065] For example, taking N=4 as an example, that is, there are four consecutive frames of face images, and the second formula above is:

[0066] in, This is the weighting function; values ​​are positive for items with the same ID and negative for items with different IDs. See below:

[0067]

[0068] It's important to note that when frames i and j represent the same person, the weights for the L2 loss differ depending on the frame interval. Adjacent frames (|ij|=1) have the highest similarity, hence a weight of 1.5; frames one frame apart have a weight of 1.2; and frames two frames apart have a weight of 1. The purpose of setting different weights is to constrain the model, making it assign smaller distances to frames with closer intervals. When frames i and j represent different people, they are not taken from a continuous usage scenario, and the frame interval is meaningless; therefore, the weights for both are -1.

[0069] Step 203b: Based on the above-mentioned second facial feature information, obtain the second loss.

[0070] For example, the second loss mentioned above can be a second L2 loss.

[0071] For example, the second L2 loss is calculated based on the third formula as the sum of the L2 losses between each pair of the second facial feature information corresponding to the N frames of the target object's face image in the input face recognition model.

[0072] For example, taking N=4 as an example, that is, there are four consecutive frames of face images, and the third formula above is:

[0073] Thus, by calculating the L2 loss on the second face feature information of the intermediate layer, constraints on the consistency of face pose in consecutive frames can be explicitly added during model training.

[0074] Step 203c: The above-mentioned classification cross-entropy loss, the above-mentioned first L2 loss, and the above-mentioned second L2 loss are weighted and calculated to obtain the target loss parameter.

[0075] For example, the first L2 loss is the sum of the L2 losses among the first facial feature information.

[0076] For example, the second L2 loss is the sum of the L2 losses among the second facial feature information.

[0077] For example, the classification cross-entropy loss, the first L2 loss, and the second L2 loss are weighted and calculated based on the fourth formula to obtain the final target loss parameter.

[0078] For example, the fourth formula above is: .

[0079] in, , These are the weighting coefficients, 0.5 and 0.2 respectively.

[0080] For example, the final target loss parameter is composed of the weighted sum of the three losses mentioned above. By backpropagating the target loss parameter, the parameters of the face recognition model are updated to obtain a new face recognition model.

[0081] Thus, by simultaneously learning the classification cross-entropy loss, the L2 loss between facial feature information of N frames of images based on contrastive learning, and the L2 loss between mid-level facial feature information of N frames of images, the aggregation degree of the face recognition model for different faces with the same user ID and the discrimination degree of faces with different user IDs are improved, and the model's ability to aggregate continuous face images is also enhanced.

[0082] In the face recognition model training method provided in this application embodiment, face images of a target object are acquired in N consecutive frames; the face images are input into the face recognition model for feature extraction to obtain first face feature information and second face feature information; the face extraction model includes X feature extraction layers; the first face feature information is obtained after sequential processing by the X feature extraction layers, and the second face feature information is obtained after processing by the i-th layer in the X feature extraction layers; i∈{2, 3, ..., 14}; based on the first face feature information and the second face feature information, a target loss parameter is calculated; based on the target loss parameter, the face recognition model is trained to obtain a trained face recognition model. In this way, by inputting multiple face images, we can obtain first face feature information and second face feature information, which are used to represent face feature information at different levels. Based on the feature information of these two, we obtain the target loss parameters of the face recognition model and train the face recognition model to obtain a trained face recognition model. Since the face recognition model obtains face feature information at different levels and calculates the loss of these face feature information, and applies the loss to train the face recognition model, the face recognition model can recognize the face feature information of different faces more accurately, thereby improving the pass rate and stability of face recognition.

[0083] This application provides a method for training a face recognition model. Figure 3 A flowchart illustrating a training method for a face recognition model provided in an embodiment of this application is shown. Figure 3 As shown, the training method for the face recognition model provided in this application embodiment may include the following steps 301 to 304.

[0084] Step 301: Obtain the face image of the current frame.

[0085] For example, the aforementioned current frame face image can be a face image acquired in real time by an electronic device.

[0086] For example, users can choose to enable the "real-time face recognition" function on their electronic devices. Once enabled, the front-facing camera of the electronic device will capture the user's face image in real time at preset time intervals (e.g., 200ms).

[0087] Step 302: Detect the key point information of the face image in the current frame.

[0088] For example, the aforementioned key points may include: left eye, right eye, nose, left corner of mouth, right corner of mouth, etc.

[0089] For example, the key point information of the current frame face image mentioned above can be the position and coordinate information of key points in the target face image.

[0090] For example, face detection is performed on the face image of the current frame to obtain the face bounding box and coordinate information of key points in the target face image.

[0091] Step 303: Based on the key point information of the target face image, process the target face image and input it into the face recognition model. After passing through the X-layer feature extraction layer in the face recognition model, process it sequentially to obtain the third face feature information.

[0092] For example, the face recognition model described above is a face recognition model trained using the training methods described in steps 201 to 204.

[0093] Step 304: Match the above-mentioned third facial feature information with the preset facial feature information and output the first recognition result.

[0094] For example, the aforementioned preset facial feature information is the facial feature information corresponding to a preset user's face in the electronic device.

[0095] For example, the first recognition result is used to indicate whether the current frame face image is the same person as the preset face image, that is, to determine whether the current user is the preset user.

[0096] For example, after matching the aforementioned third facial feature information with preset facial feature information, a first distance threshold is obtained, and the first distance threshold is compared with the preset distance threshold.

[0097] For example, the first distance threshold is used to characterize the degree of difference between the third facial feature information and the preset facial feature information.

[0098] Specifically, the first distance threshold can be expressed as d(f i f enroll ), where f i For third-party facial feature information, f enroll Preset facial feature information.

[0099] For example, the aforementioned preset distance threshold is a custom setting of the electronic device. Further, this preset distance threshold can be expressed as D. th1 .

[0100] In one possible embodiment, the first distance threshold is compared with a preset distance threshold. If the first distance threshold is less than the preset distance threshold, the electronic device outputs the result of successful recognition of the target face image corresponding to the third face feature information, that is, the recognition result indicates that the face verification is successful.

[0101] In another possible embodiment, the first distance threshold is compared with a preset distance threshold. If the first distance threshold is greater than the preset distance threshold, that is, the recognition result indicates that the face verification has failed, then step 401 is continued.

[0102] In the face recognition method based on a face recognition model provided in this application embodiment, a face image of the current frame is acquired; key point information of the face image of the current frame is detected; the face image of the current frame is processed based on the key point information and input into a face recognition model, which is then processed sequentially through the X-layer feature extraction layer of the face recognition model to obtain third face feature information; the third face feature information is matched with preset face feature information to output a first recognition result; wherein, the face recognition model is a face recognition model trained using the training method of the face recognition model in the first aspect. Thus, by using the face recognition model trained by the training method of the face recognition model in this application embodiment to extract face image features, the success rate and stability of face recognition are improved.

[0103] Optionally, in this embodiment of the application, after step 304 "matching the above-mentioned third facial feature information with preset facial feature information and outputting the first recognition result", the facial recognition method based on the facial recognition model provided in this embodiment of the application further includes:

[0104] Step 401: If the first recognition result indicates that the face verification has failed, and if it is determined that the key point information is missing, then the third face feature information is matched with the face feature information corresponding to any face image in the previous Y frames of the current frame, and the second recognition result is output.

[0105] For example, any of the face images in the first Y frames mentioned above can be images captured in real time by an electronic device.

[0106] For example, any face image in the preceding Y frames may belong to the same user as the face image in the current frame, or they may not belong to the same user.

[0107] Optionally, in this embodiment of the application, step 401 above includes steps 401a to 401c:

[0108] Step 401a: If the first recognition result indicates that the face verification has failed, detect the number of key point information in the current frame face image; and determine the display status of the current frame face image based on the number of key point information.

[0109] For example, the display state of the current frame face image may include: abnormal display and normal display.

[0110] As further exemplarily, the above-mentioned abnormal display state may include at least one of the following: large area of ​​the face is obscured, the face is rotated at a large angle, or there is severe half-face distortion.

[0111] In one possible embodiment, if the number of key point information points detected is greater than a predetermined number, and the display status of the current frame face image is determined to be normal, since the first distance threshold is greater than the preset distance threshold, the electronic device outputs the result that the target face image corresponding to the third face feature information fails to be recognized, i.e., face verification fails.

[0112] In one possible embodiment, if the number of key point information points detected is less than a predetermined number, and it is determined that the display state of the current frame face image is abnormal, the electronic device outputs the result that the current frame face image corresponding to the third face feature information fails to be recognized, that is, the face verification fails; if it is determined that the display state of the current frame face image is normal, it indicates that the face image information is missing a lot, and thus the process continues to step 401b.

[0113] Step 401b: If too many face images are missing in the current frame, compare the third face feature information with the fourth face feature information of the face images in the previous m frames to obtain a second distance threshold, and compare the second distance threshold with a preset distance threshold.

[0114] For example, the aforementioned first m frames of face images are any one of the aforementioned first Y frames of face images.

[0115] For example, the previous m frames of face images are face images that have already passed face recognition before the face image of the current frame. It should be noted that when searching for the frames that have already passed (i.e., the face images of the previous m frames of the current frame), the search range should be as small as possible. The search range can be set by the user or it can be the default range of the electronic device, such as 1 second, that is, the previous 5 frames.

[0116] For example, the second distance threshold is used to characterize the degree of difference between the third facial feature information and the fourth facial feature information.

[0117] Specifically, the second distance threshold can be expressed as d(f i , f pre ), where f i For third-party facial feature information, f pre Preset facial feature information

[0118] In one possible embodiment, the second distance threshold is compared with a preset distance threshold. If the second distance threshold is less than the preset distance threshold, the electronic device outputs the result that the target face image corresponding to the third face feature information has passed the recognition, that is, the face verification is successful.

[0119] In another possible embodiment, the second distance threshold is compared with a preset distance threshold. If the second distance threshold is greater than the preset distance threshold, then step 401c is continued.

[0120] Step 401c: Determine whether the previous frame face image of the current frame face image has passed face recognition. Based on the recognition result of the previous frame face image, output the recognition result of the current frame face image.

[0121] In one possible embodiment, if the previous frame of the target face image fails the face recognition result, the electronic device outputs the result that the target face image corresponding to the third face feature information fails to be recognized, that is, the face verification fails.

[0122] In another possible embodiment, the face image of the previous frame of the current frame is compared with the face image of the previous frame through the face recognition result. If the change of the key point position in the current frame face image is small, it indicates that the face position has not changed much, and the electronic device outputs the result of face recognition passing, that is, face verification is successful; otherwise, it outputs the result of face recognition failing, that is, face verification fails.

[0123] For example, such as Figure 4 As shown, test images t1, t2, and t3 are consecutive test images. The feature distance between test image t1 (i.e., the current frame face image) and the registration image (i.e., the preset face image) is 0.46 (i.e., the first distance threshold), which is less than the recognition distance threshold of 0.47 (and the preset distance threshold), so it can be identified as a user image. However, when test images t2 and t3 are directly compared with the registration image, the distance is greater than the threshold, so they will fail to be recognized (it should be noted that test images t2 and t3 at this time are the current frame face images). Since consecutive frames have a strong correlation, test image t2 can be compared with the previously passed test image t1, and its distance is 0.46, so it can pass. Similarly, test image t3 can be compared with the previously passed image t2, and its distance is 0.458, so it can pass (it should be noted that test images t2 and t3 at this time are the face images of the previous m frames).

[0124] In this way, facial feature information is extracted through a facial recognition model, and then the facial feature information is compared with preset facial feature information and facial feature information of faces that have passed the previous frame. In addition, the changes in the facial position state are combined to obtain the final verification result, thereby reducing the user's face rejection rate and improving the user experience.

[0125] It should be noted that the face recognition model training method provided in this application embodiment can be executed by a face recognition model training device, an electronic device, or a functional module or entity within an electronic device. This application embodiment uses a face recognition model training device executing the face recognition model training method as an example to illustrate the face recognition model training device provided in this application embodiment.

[0126] Figure 5 This diagram illustrates a possible structural schematic of a training device for a face recognition model involved in an embodiment of this application. For example... Figure 5 As shown, the training device 700 for the face recognition model may include: an acquisition module 601 and a processing module 602; the acquisition module 601 is used to acquire face images of a target object in N consecutive frames; the processing module 602 is used to input the aforementioned face images of the target object in N consecutive frames into the face recognition model for feature extraction to obtain first face feature information and second face feature information; the face extraction model includes X feature extraction layers; the aforementioned first face feature information is obtained after being processed sequentially by the aforementioned X feature extraction layers, and the aforementioned second face feature information is obtained after being processed by the i-th layer in the aforementioned X feature extraction layers; i∈{2, 3, ..., X-1}; the aforementioned processing module 602 is also used to calculate a target loss parameter based on the aforementioned first face feature information and the aforementioned second face feature information; the aforementioned processing module 602 is also used to train the face recognition model based on the aforementioned target loss parameter to obtain a trained face recognition model; wherein, N and X are positive integers greater than 1.

[0127] Optionally, in this embodiment of the application, the processing module 602 is specifically used to: obtain a classification cross-entropy loss and a first loss based on the first face feature information; obtain a second loss based on the second face feature information; perform a weighted calculation on the classification cross-entropy loss, the first loss, and the second loss to obtain a target loss parameter; the first loss is the sum of the losses among the first face feature information; the second loss is the sum of the losses among the second face feature information.

[0128] Optionally, in this embodiment of the application, the processing module 602 is specifically used to: input the face image into the X-layer feature extraction layer for sequential processing to obtain a first face feature image; process the first face feature image through convolution transformation to obtain the first face feature information; and process the second face feature image obtained after processing by the i-th layer feature extraction layer through a convolution layer to obtain the second face feature information.

[0129] Optionally, in this embodiment of the application, the processing module 602 is specifically used to: map the first face feature information to M preset face recognition labels through a fully connected layer to obtain face recognition prediction parameters, which are used to indicate the predicted face recognition labels; calculate the classification cross-entropy loss between the predicted face recognition label and the M preset face recognition labels; and calculate the first loss based on the first face feature information.

[0130] In the face recognition model training device provided in this application embodiment, N consecutive frames of face images of the target object are acquired; the face images are input into the face recognition model for feature extraction to obtain first face feature information and second face feature information; the face extraction model includes X feature extraction layers; the first face feature information is obtained after sequential processing by the X feature extraction layers, and the second face feature information is obtained after processing by the i-th layer in the X feature extraction layers; i∈{2, 3, ..., 14}; based on the first face feature information and the second face feature information, a target loss parameter is calculated; based on the target loss parameter, the face recognition model is trained to obtain a trained face recognition model. In this way, by inputting multiple face images, we can obtain first face feature information and second face feature information, which are used to represent face feature information at different levels. Based on the feature information of these two, we obtain the target loss parameters of the face recognition model and train the face recognition model to obtain a trained face recognition model. Since the face recognition model obtains face feature information at different levels and calculates the loss of these face feature information, and applies the loss to train the face recognition model, the face recognition model can recognize the face feature information of different faces more accurately, thereby improving the pass rate and stability of face recognition.

[0131] Figure 6 This illustration shows a possible structural diagram of a face recognition device based on a face recognition model, as described in an embodiment of this application. Figure 6 As shown, the face recognition device 700 based on the face recognition model may include: an acquisition module 701, a detection module 702, and a processing module 703; the acquisition module 701 is further used to acquire a target face image; the detection module 702 is used to detect key point information of the current frame face image; the processing module 703 is further used to process the current frame face image based on the key point information and input it into the face recognition model, and after sequential processing through the X-layer feature extraction layer, obtain third face feature information; the third face feature information is matched with preset face feature information, and a first recognition result is output; wherein, the face recognition model is a face recognition model trained using the training method of the face recognition model.

[0132] In the face recognition device based on a face recognition model provided in this application embodiment, a current frame face image is acquired; key point information of the current frame face image is detected; the current frame face image is processed based on the key point information and input into a face recognition model. After passing through the X-layer feature extraction layer of the face recognition model, third face feature information is obtained; the third face feature information is matched with preset face feature information, and a first recognition result is output; wherein, the face recognition model is a face recognition model trained using the training method of the face recognition model in the first aspect. Thus, by using the face recognition model trained by the face recognition model training method of this application embodiment to extract face image features, the success rate and stability of face recognition are improved.

[0133] The training device for the face recognition model and the face recognition device based on the face recognition model in this application embodiment can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be an electronic device, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0134] The training device for the face recognition model and the face recognition device based on the face recognition model in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit them.

[0135] The face recognition model training device and the face recognition device based on the face recognition model provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0136] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-mentioned face recognition model training method and face recognition method embodiment based on the face recognition model, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0137] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0138] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0139] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0140] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0141] The processor 110 is configured to acquire N consecutive frames of facial images of a target object; the processor 110 is also configured to input the aforementioned N consecutive frames of facial images of the target object into a face recognition model for feature extraction to obtain first facial feature information and second facial feature information; the face extraction model includes X feature extraction layers; the aforementioned first facial feature information is obtained after sequential processing by the aforementioned X feature extraction layers, and the aforementioned second facial feature information is obtained after processing by the i-th layer of the aforementioned X feature extraction layers; i∈{2, 3, ..., X-1}; the processor 110 is also configured to calculate a target loss parameter based on the aforementioned first facial feature information and the aforementioned second facial feature information; the processor 110 is also configured to train the face recognition model based on the aforementioned target loss parameter to obtain a trained face recognition model; wherein N and X are positive integers greater than 1.

[0142] Optionally, in this embodiment of the application, the processor 110 is specifically used to: obtain a classification cross-entropy loss and a first loss based on the first face feature information; obtain a second loss based on the second face feature information; perform a weighted calculation on the classification cross-entropy loss, the first loss, and the second loss to obtain a target loss parameter; the first loss is the sum of the losses among the first face feature information; the second loss is the sum of the losses among the second face feature information.

[0143] Optionally, in this embodiment of the application, the processor 110 is specifically used to: input the face image into the X-layer feature extraction layer for sequential processing to obtain a first face feature image; process the first face feature image through convolution transformation to obtain the first face feature information; and process the second face feature image obtained after processing by the i-th layer feature extraction layer through a convolution layer to obtain the second face feature information.

[0144] Optionally, in this embodiment of the application, the processor 110 is specifically used to: map the first facial feature information to M preset facial recognition labels through a fully connected layer to obtain facial recognition prediction parameters, which are used to indicate the predicted facial recognition labels; calculate the classification cross-entropy loss between the predicted facial recognition label and the M preset facial recognition labels; and calculate the first loss based on the first facial image feature information.

[0145] In the electronic device provided in this application embodiment, N consecutive frames of facial images of a target object are acquired; the facial images are input into a facial recognition model for feature extraction to obtain first facial feature information and second facial feature information; the facial extraction model includes X feature extraction layers; the first facial feature information is obtained after sequential processing by the X feature extraction layers, and the second facial feature information is obtained after processing by the i-th layer in the X feature extraction layers; i∈{2, 3, ..., 14}; based on the first facial feature information and the second facial feature information, a target loss parameter is calculated; based on the target loss parameter, the facial recognition model is trained to obtain a new facial recognition model. Thus, by inputting multiple face images, we can obtain first and second face feature information, two types of face feature information used to represent different levels of face features. Based on these two feature information, we obtain the target loss parameters of the face recognition model and train the face recognition model to obtain a new face recognition model. Since the face recognition model obtains face feature information at different levels and calculates the loss of these face feature information, and applies the loss to train the face recognition model, the face recognition model can recognize the face feature information of different faces more accurately, thereby improving the pass rate and stability of face recognition.

[0146] Optionally, in this embodiment, the processor 110 is further configured to acquire a face image of the current frame; the processor 110 is configured to detect key point information of the face image of the current frame; the processor 110 is further configured to process the target face image based on the key point information and input it into the face recognition model, and process it sequentially through the X-layer feature extraction layer to obtain third face feature information; the processor 110 is further configured to match the third face feature information with preset face feature information and output a first recognition result; wherein, the face recognition model is a face recognition model trained using the training method of the face recognition model.

[0147] In the electronic device provided in this application embodiment, a current frame face image is acquired; key point information of the current frame face image is detected; the current frame face image is processed based on the key point information and input into a face recognition model. After passing through the X-layer feature extraction layer of the face recognition model, third face feature information is obtained; the third face feature information is matched with preset face feature information, and a first recognition result is output. The face recognition model is trained using the training method of the face recognition model in the first aspect. Thus, by using the face recognition model trained by the training method of the face recognition model in this application embodiment to extract face image features, the success rate and stability of face recognition are improved.

[0148] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0149] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0150] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0151] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described face recognition model training method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0152] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0153] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described face recognition model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0154] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0155] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the training method embodiment of the face recognition model described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be an electronic device, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0158] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training a face recognition model, characterized in that, The method includes: Acquire face images of a target object in N consecutive frames; The face image is input into a face recognition model for feature extraction to obtain first face feature information and second face feature information; the face recognition model includes X feature extraction layers; the first face feature information is obtained after being processed sequentially by the X feature extraction layers, and the second face feature information is obtained after being processed by the i-th layer in the X feature extraction layers; i∈{2,3,……X-1}; Based on the first facial feature information and the second facial feature information, the target loss parameters are calculated; Based on the target loss parameters, the face recognition model is trained to obtain a trained face recognition model; Where N and X are positive integers greater than 1; The step of calculating the target loss parameters based on the first facial feature information and the second facial feature information includes: Based on the first facial feature information, the classification cross-entropy loss and the first loss are obtained; Based on the second facial feature information, a second loss is obtained; The target loss parameter is obtained by weighting the classification cross-entropy loss, the first loss, and the second loss; the first loss is the sum of the losses among the first facial feature information. The second loss is the sum of the losses among the second facial feature information.

2. The method according to claim 1, characterized in that, The step of inputting the face image into a face recognition model for feature extraction to obtain first face feature information and second face feature information includes: The face image is input into the X-layer feature extraction layer for sequential processing to obtain the first face feature image; The first face feature image is processed by convolution transformation to obtain the first face feature information; The second face feature image obtained after processing by the i-th feature extraction layer is processed by a convolutional layer to obtain the second face feature information.

3. The method according to claim 1, characterized in that, The step of obtaining the classification cross-entropy loss and the first loss based on the first facial feature information includes: The first facial feature information is mapped to M preset facial recognition labels through a fully connected layer to obtain facial recognition prediction parameters, which are used to indicate the predicted facial recognition labels. Calculate the classification cross-entropy loss between the predicted face recognition label and the M preset face recognition labels; Based on the first facial feature information, the first loss is calculated and processed.

4. A face recognition method based on a face recognition model, characterized in that, The method includes: Get the face image of the current frame; Detect the key point information of the current frame face image; The current frame face image is processed based on the key point information and input into the face recognition model. After being processed sequentially by the X-layer feature extraction layer, the third face feature information is obtained. The third facial feature information is matched with the preset facial feature information to output the first recognition result; The face recognition model is a face recognition model trained using the training method described in any one of claims 1 to 3.

5. A training device for a face recognition model, characterized in that, The training device for the face recognition model includes: an acquisition module and a processing module; The acquisition module is used to acquire face images of the target object in N consecutive frames; The processing module is used to input the face image acquired by the acquisition module into a face recognition model for feature extraction to obtain first face feature information and second face feature information; the face recognition model includes X feature extraction layers; the first face feature information is obtained after sequential processing by the X feature extraction layers, and the second face feature information is obtained after processing by the i-th layer in the X feature extraction layers; i∈{2, 3, ..., X-1}; The processing module is further configured to calculate the target loss parameters based on the first facial feature information and the second facial feature information; The processing module is further configured to train the face recognition model based on the target loss parameters to obtain a trained face recognition model; Where N and X are positive integers greater than 1; The processing module is specifically used for: Based on the first facial feature information, the classification cross-entropy loss and the first loss are obtained; Based on the second facial feature information, a second loss is obtained; The target loss parameter is obtained by weighting the classification cross-entropy loss, the first loss, and the second loss; the first loss is the sum of the losses among the first facial feature information. The second loss is the sum of the losses among the second facial feature information.

6. The apparatus according to claim 5, characterized in that, include: The processing module is specifically used for: The face image is input into the X-layer feature extraction layer for sequential processing to obtain the first face feature image; The first face feature image is processed by convolution transformation to obtain the first face feature information; The second face feature image obtained after processing by the i-th feature extraction layer is processed by a convolutional layer to obtain the second face feature information.

7. The apparatus according to claim 5, characterized in that, include: The processing module is specifically used for: The first facial feature information is mapped to M preset facial recognition labels through a fully connected layer to obtain facial recognition prediction parameters, which are used to indicate the predicted facial recognition labels. Calculate the classification cross-entropy loss between the predicted face recognition label and the M preset face recognition labels; Based on the first facial feature information, the first loss is calculated and processed.

8. A face recognition device based on a face recognition model, characterized in that, The device includes: The module includes an acquisition module, a detection module, and a processing module. The acquisition module is used to acquire the face image of the current frame; The detection module is used to detect the key point information of the current frame face image acquired by the acquisition module; The processing module is used to process the current frame face image obtained by the acquisition module based on the key point information detected by the detection module, and input it into the face recognition model. After being processed sequentially by the X-layer feature extraction layer, the third face feature information is obtained. The processing module is used to match the third facial feature information with preset facial feature information and output a first recognition result; The face recognition model is a face recognition model trained using the training method described in any one of claims 1 to 3.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the training method for the face recognition model as described in any one of claims 1 to 3, or the face recognition method based on the face recognition model as described in claim 4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the training method for the face recognition model as described in any one of claims 1 to 3, or the face recognition method based on the face recognition model as described in claim 4.

Images