A face recognition method and system based on a skeleton calibration algorithm

Abstract

The application provides a face recognition method and system based on a skeleton calibration algorithm, and belongs to the technical field of face recognition. The method can determine each first three-dimensional skeleton coordinate sequence and each first pixel point feature information of a to-be-recognized object within a preset time based on a face recognition video stream; determine a second three-dimensional skeleton coordinate sequence according to the first three-dimensional skeleton coordinate sequence, the first pixel point feature information and a preset adaptive distillation model. According to a preset coupling degree calculation formula, each first three-dimensional skeleton coordinate sequence and each first pixel point feature information, the coupling degree values corresponding to each selected pixel point in a preset region are calculated to determine the corresponding multi-modal feature aggregation amount; based on each multi-modal feature aggregation amount, the second three-dimensional skeleton coordinate sequence and a preset attention mechanism, a skeleton pixel feature vector is determined, and based on the skeleton pixel feature vector and a preset face recognition database, a face recognition result of the to-be-recognized object is determined.

Description

Technical Field

[0001] This application relates to the field of face recognition technology, and in particular to a face recognition method and system based on a skeleton calibration algorithm. Background Technology

[0002] With the development of artificial intelligence technology, facial recognition has been widely used in security monitoring, access control, and financial payments. Traditional facial recognition methods mainly rely on two-dimensional image features, extracting information such as facial texture and contours for identity verification.

[0003] However, in real-world applications, facial recognition often faces numerous challenges, such as complex lighting conditions, changes in posture, and occlusion (e.g., wearing masks, sunglasses, hats, etc.). These factors can lead to incomplete or altered facial features, severely impacting the accuracy and stability of recognition.

[0004] To address the occlusion problem, some existing technologies attempt to directly extract and complete features from occluded regions using deep learning models. However, these methods often rely on large amounts of labeled data and lack generalization ability in complex occlusion scenarios. Furthermore, in practical applications, manual intervention may be required for recognition, incurring significant labor costs.

[0005] Therefore, there is an urgent need for a technical solution that can perform more flexible, accurate and efficient face recognition in complex occlusion scenarios. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a face recognition method and system based on a skeletal calibration algorithm, which enables more flexible, accurate, and efficient face recognition in complex occlusion scenarios.

[0007] On one hand, embodiments of this application provide a face recognition method based on a skeletal calibration algorithm, the method comprising:

[0008] Based on the face recognition video stream from the preset face recognition device group, determine the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be recognized within a preset time period;

[0009] Based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and the preset adaptive distillation model, a corresponding second three-dimensional skeleton coordinate sequence is determined; wherein, the second three-dimensional skeleton coordinate sequence is the three-dimensional skeleton coordinate sequence after completing the occluded face of the object to be identified;

[0010] Based on the preset coupling degree calculation formula, the coordinate sequences of each first three-dimensional skeleton, and the feature information of each first pixel, the coupling degree value corresponding to each selected pixel in the preset area is calculated. The corresponding multimodal feature aggregation amount is determined based on the weighted calculation result of each coupling degree value, the feature information of the first pixel, and the coordinate sequence of the second three-dimensional skeleton. The coupling degree value is used to characterize the dynamic correlation between the skeleton and the pixel.

[0011] Based on the aggregated multimodal features, the second three-dimensional skeletal coordinate sequence, and the preset attention mechanism, the skeletal pixel feature vector is determined, and the face recognition result of the object to be recognized is determined according to the skeletal pixel feature vector and the preset face recognition database.

[0012] In one implementation of this application, based on a face recognition video stream from a preset face recognition device group, the first three-dimensional skeletal coordinate sequence of the object to be recognized and the feature information of each first pixel point within a preset time period are determined, specifically including:

[0013] The face recognition video stream from the preset face recognition device group is input into a pre-trained skeletal calibration model to determine the coordinates of several approximate skeletal points in the face recognition video stream corresponding to the object to be recognized based on the model output results.

[0014] Map the coordinates of each of the approximate bone points to a preset depth map coordinate system, and add the mapped bone coordinate points to each of the first three-dimensional bone coordinate sequences in chronological order.

[0015] Determine the RGB images of each face in the face recognition video stream; wherein each frame of the face recognition video stream contains an RGB image and a depth image;

[0016] According to a preset grid, each of the facial RGB images is divided into grids to determine the feature vectors corresponding to each of the facial RGB images as the feature information of the second pixel point;

[0017] Based on the coordinate sequences of each of the first three-dimensional skeletons and the feature information of each of the second pixels, a corresponding skeleton pixel correlation coefficient matrix is ​​determined, so as to determine the feature information of the first pixel based on the skeleton pixel correlation coefficient matrix, the preset correlation coefficient threshold and the feature information of the second pixels.

[0018] In one implementation of this application, a corresponding bone pixel correlation coefficient matrix is ​​determined based on each of the first three-dimensional bone coordinate sequences and each of the second pixel feature information, specifically including:

[0019] Based on the first three-dimensional skeleton coordinate sequence and the second pixel feature information corresponding to the same frame, the dynamic skeleton pixel correlation coefficient is determined, and a set of dynamic skeleton pixel correlation coefficients corresponding to each video frame is constructed; the dynamic skeleton pixel correlation coefficient is calculated based on the distance between the skeleton mapping coordinate point and the corresponding grid center;

[0020] The dynamic skeleton pixel correlation coefficient set is matched with a preset standard correlation coefficient library to determine the skeleton pixel correlation coefficient matrix based on the matching result; wherein, the preset standard correlation coefficient set is constructed based on the unoccluded face recognition video stream of the pre-recorded object.

[0021] In one implementation of this application, the first pixel feature information is determined based on the skeletal pixel correlation coefficient matrix, a preset correlation coefficient threshold, and the second pixel feature information, specifically including:

[0022] Traverse the bone pixel correlation coefficient matrix and filter correlation coefficients that are greater than the preset correlation coefficient threshold.

[0023] The pixel corresponding to the filtering correlation coefficient in the second pixel feature information is determined as the first pixel, so as to determine the first pixel feature information based on the first pixel obtained by filtering.

[0024] In one implementation of this application, a corresponding second three-dimensional skeleton coordinate sequence is determined based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and a preset adaptive distillation model, specifically including:

[0025] The first three-dimensional skeletal coordinate sequence and the first pixel feature information are input into the preset adaptive distillation model to match the initial complete skeletal coordinate sequence in the standard face template library through the pre-trained teacher model.

[0026] The student model is used to perform convolution processing on the preprocessed feature information of the first pixel and the first three-dimensional skeleton coordinate sequence to determine the three-dimensional skeleton coordinate sequence to be determined; the student model is trained based on several three-dimensional skeleton coordinate sequence samples of occluded faces.

[0027] The knowledge distillation loss function value is calculated based on the initial complete skeleton coordinate sequence and the undetermined three-dimensional skeleton coordinate sequence. If the knowledge distillation loss function value is less than a preset threshold, the undetermined three-dimensional skeleton coordinate sequence is used as the second three-dimensional skeleton coordinate sequence.

[0028] In one implementation of this application, the coupling degree value corresponding to each selected pixel within a preset region is calculated based on a preset coupling degree calculation formula, the coordinate sequence of each of the first three-dimensional skeletons, and the feature information of each of the first pixels. Specifically, this includes:

[0029] The displacement variance of each bone point is calculated based on each of the first three-dimensional bone coordinate sequences, and the reciprocal of the displacement variance of the bone point is used as the temporal stability value of the bone point. The texture entropy of the pixel point is calculated based on the feature information of each of the first pixel points.

[0030] Based on the coordinate sequences of each of the first three-dimensional skeletons and the feature information of the first pixel, the Euclidean distance between each skeleton and each of the selected pixels is determined; wherein the correlation coefficient of the selected pixels is greater than the preset correlation coefficient threshold.

[0031] The temporal stability value of the skeletal point, the texture entropy of the pixel point, and the Euclidean distance value are input into the preset coupling degree calculation formula to calculate the coupling degree value of the selected pixel point.

[0032] In one implementation of this application, the corresponding multimodal feature aggregation amount is determined based on the weighted calculation result of each coupling degree value, the first pixel feature information, and the second three-dimensional skeleton coordinate sequence, specifically including:

[0033] Based on the feature information of each first pixel and the preset occlusion ratio prediction model, the facial occlusion ratio of the object to be identified is determined, and the pixel feature weight and the skeletal feature weight are determined based on the facial occlusion ratio; the sum of the pixel feature weight and the skeletal feature weight is 1.

[0034] The feature information of each selected pixel in the first pixel feature information is vector-encoded to obtain a first feature vector, and the feature attributes corresponding to each bone point in the second three-dimensional bone coordinate sequence are vector-encoded to determine a second feature vector; the feature attributes corresponding to the bone point include at least position encoding, motion speed and posture.

[0035] Based on the pixel feature weights and the bone feature weights, calculate the weighted sum of the first feature vector and the second feature vector of the kth bone point, and use the weighted sum as the kth first component; k is the total number of the selected pixels, and k is a natural number;

[0036] Calculate the product of the coupling degree value of the kth first component and the kth skeleton point as the kth second component, and calculate the sum of the k second components to obtain the multimodal feature aggregation amount of the corresponding selected pixel point.

[0037] In one implementation of this application, the bone pixel feature vector is determined based on the aggregated multimodal features, the second three-dimensional skeleton coordinate sequence, and a preset attention mechanism, specifically including:

[0038] The aggregated multimodal features are concatenated with the second three-dimensional skeleton coordinate sequence to obtain a concatenated vector;

[0039] The concatenated vector is input into the preset attention mechanism to map the query vector, key vector, and value vector, and the attention mechanism output value corresponding to each attention head is calculated.

[0040] The output values ​​of each attention mechanism are concatenated and linearly transformed to obtain the skeletal pixel feature vector.

[0041] In one implementation of this application, determining the face recognition result of the object to be recognized based on the skeletal pixel feature vector and a preset face recognition database specifically includes:

[0042] According to the preset Bayesian decision rule, the posterior probability between the skeletal pixel feature vector and each identity entry information in the preset face recognition database is calculated respectively.

[0043] The maximum posterior probability value among the posterior probabilities is determined, and the identity identifier corresponding to the maximum posterior probability value is used as the face recognition result of the object to be identified.

[0044] On the other hand, embodiments of this application also provide a face recognition system based on a skeletal calibration algorithm, the system comprising:

[0045] The first determining module is used to determine the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be identified within a preset time period based on the face recognition video stream from the preset face recognition device group.

[0046] The second determining module is used to determine a corresponding second three-dimensional skeleton coordinate sequence based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and a preset adaptive distillation model; wherein, the second three-dimensional skeleton coordinate sequence is a three-dimensional skeleton coordinate sequence after completing the occluded face of the object to be identified.

[0047] The calculation module is used to calculate the coupling degree value corresponding to each selected pixel in a preset area according to the preset coupling degree calculation formula, the coordinate sequence of each first three-dimensional skeleton and the feature information of each first pixel, and to determine the corresponding multimodal feature aggregation amount based on the weighted calculation result of each coupling degree value, the feature information of the first pixel and the coordinate sequence of the second three-dimensional skeleton; the coupling degree value is used to characterize the dynamic correlation between the skeleton and the pixel.

[0048] The third determining module is used to determine the bone pixel feature vector based on the aggregated amount of each multimodal feature, the second three-dimensional bone coordinate sequence and the preset attention mechanism, and to determine the face recognition result of the object to be recognized based on the bone pixel feature vector and the preset face recognition database.

[0049] Compared with the prior art, the significant advantages of this application are as follows:

[0050] The above scheme, through a pre-set adaptive distillation model and the collaboration of teacher and student models, completes the second three-dimensional skeletal coordinate sequence of the occluded face based on the first three-dimensional skeletal coordinate sequence and the feature information of the first pixel. This provides complete skeletal feature information for accurate recognition, effectively overcoming the interference of occlusion on recognition, improving recognition accuracy in complex occlusion scenarios, and handling different degrees of occlusion and scenarios, thus enhancing the generalization ability of face recognition. Simultaneously, the coupling degree value is calculated to represent the dynamic relationship between bones and pixels, fully exploring the intrinsic connections between multimodal data to calculate the multimodal feature aggregation, making feature expression more comprehensive and enhancing recognition accuracy. This eliminates the need for manual intervention, reducing labor costs. Therefore, it solves the problems of low accuracy, poor stability, weak generalization ability, and high labor costs in face recognition under complex occlusion scenarios. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0052] Figure 1 This is a flowchart illustrating a face recognition method based on a skeleton calibration algorithm in an embodiment of this application.

[0053] Figure 2 This is a schematic diagram of the structure of a face recognition system based on a skeleton calibration algorithm in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] This application provides a face recognition method and system based on a skeleton calibration algorithm to solve the technical problem that the face recognition generalization ability is insufficient in complex occlusion scenarios, making it difficult to perform flexible, accurate and efficient recognition, as well as the high manpower cost.

[0056] The various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] This application provides a face recognition method based on a skeleton calibration algorithm, such as... Figure 1 As shown, the method may include steps S101-S104:

[0058] S101, the server determines the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be identified within a preset time period based on the face recognition video stream from the preset face recognition device group.

[0059] It should be noted that the server, as the execution entity of the face recognition method based on the skeleton calibration algorithm, exists only as an example, and the execution entity is not limited to the server. This application does not make any specific limitation in this regard.

[0060] The pre-set facial recognition device group includes at least an RGB camera and a depth camera. The facial recognition video stream can be understood as a video clip captured of the face of the object to be recognized within a preset time period. This application can be applied to access control recognition scenarios such as hospitals and laboratories. The skeletal points corresponding to the first three-dimensional skeletal coordinate sequence include at least one or more of the following: frontal bone, eye socket, nasal bone, zygomatic bone, mandible, and occlusal line. This application can pre-train a convolutional neural network model, which is obtained by training several RGB images and depth images of labeled skeletal points. Thus, skeletal points are recognized through the facial recognition video stream and the convolutional neural network model.

[0061] In this embodiment of the application, based on the face recognition video stream from a preset face recognition device group, the first three-dimensional skeletal coordinate sequence of the object to be recognized and the feature information of each first pixel point within a preset time period are determined, specifically including:

[0062] A face recognition video stream from a pre-set face recognition device group is input into a pre-trained skeletal calibration model. Based on the model output, the coordinates of several approximate skeletal points corresponding to the object to be recognized in the face recognition video stream are determined. Each approximate skeletal point coordinate is mapped to a pre-set depth map coordinate system, and the mapped coordinates are added sequentially to each first three-dimensional skeletal coordinate sequence according to time order. Each facial RGB image in the face recognition video stream is determined. Each frame of the face recognition video stream contains an RGB image and a depth image. According to a pre-set grid, each facial RGB image is divided into grids to determine the feature vectors corresponding to each facial RGB image as the second pixel feature information. Based on each first three-dimensional skeletal coordinate sequence and each second pixel feature information, a corresponding skeletal pixel correlation coefficient matrix is ​​determined. Based on the skeletal pixel correlation coefficient matrix, a pre-set correlation coefficient threshold, and the second pixel feature information, the first pixel feature information is determined.

[0063] In other words, this application pre-trains a skeletal calibration model, which is a machine learning model trained on a large amount of data. This model is used to analyze and detect the coordinates of several approximate skeletal points corresponding to the object to be identified from the face recognition video stream. For example, the model can determine the coordinates of these approximate skeletal points by identifying some key facial feature points, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. These coordinates reflect the approximate position information of the facial bones in the video stream. Subsequently, the obtained coordinates of each approximate skeletal point are mapped to a preset depth map coordinate system. The depth map coordinate system is a coordinate system used to represent the depth information of an object in three-dimensional space. Through this mapping, the position of the skeletal points can be combined with the depth information. Then, in chronological order, the mapped skeletal coordinate points are added to each first three-dimensional skeletal coordinate sequence. Because the face recognition video stream consists of a series of continuous frames, the position of the skeletal points will change over time. Therefore, recording these coordinate points in chronological order forms a first three-dimensional skeletal coordinate sequence that reflects the movement and positional changes of the bones over a period of time.

[0064] Subsequently, the server can extract RGB images, including facial RGB images containing color and texture information. Then, based on a preset grid, each facial RGB image is divided into grid regions. The preset grid is a predefined partitioning method that divides the facial RGB image into multiple small grid regions. For each grid region, a feature vector is determined using a specific algorithm (such as extracting the color and texture features of pixels within that region). These feature vectors constitute the second pixel feature information. In this way, the features of the facial RGB image can be represented in a structured manner, facilitating subsequent analysis and processing. Furthermore, this application can establish a connection between skeletal information and pixel feature information, constructing a skeletal-pixel correlation coefficient matrix. Using this matrix, a preset correlation coefficient threshold, and the second pixel feature information, the first pixel feature information is determined. The preset correlation coefficient threshold is a threshold set by the user based on the actual usage scenario; this application does not specifically limit it.

[0065] Furthermore, the above-mentioned determination of the corresponding bone pixel correlation coefficient matrix based on each first three-dimensional bone coordinate sequence and each second pixel feature information specifically includes:

[0066] Based on the first three-dimensional skeleton coordinate sequence and the second pixel feature information corresponding to the same frame, dynamic skeleton pixel correlation coefficients are determined, and a set of dynamic skeleton pixel correlation coefficients for each video frame is constructed. The dynamic skeleton pixel correlation coefficients are calculated based on the distance between the skeleton mapping coordinate point and the corresponding grid center. The set of dynamic skeleton pixel correlation coefficients is matched with a preset standard correlation coefficient library to determine the skeleton pixel correlation coefficient matrix based on the matching results. The preset standard correlation coefficient library is constructed based on the unoccluded face recognition video stream of pre-recorded objects.

[0067] In other words, this application calculates the distance between the bone mapping coordinate point and the grid center for the first three-dimensional skeleton coordinate sequence and the second pixel feature information corresponding to the same frame in the video stream. This distance can be Euclidean distance. For example, the Euclidean distance between the bone mapping coordinate point P(x1,y1,z1) in the first three-dimensional skeleton coordinate sequence and a certain grid center Q(x2,y2,z2) in the first pixel feature information is: The Euclidean distance is used as the dynamic skeletal pixel correlation coefficient for P(x1,y1,z1) and Q(x2,y2,z2). A dynamic skeletal pixel correlation coefficient set is constructed by calculating the dynamic skeletal pixel correlation coefficients between all skeletal points and all grid centers. Then, this application matches the constructed dynamic skeletal pixel correlation coefficient set with a preset standard correlation coefficient library. This preset standard correlation coefficient library is constructed based on pre-recorded unoccluded face recognition video streams of objects. When constructing this library, the correlation coefficients between bones and pixels in the unoccluded video stream are also calculated. These correlation coefficients reflect the correlation pattern between facial bones and pixels in the normal (unoccluded) state. This library can be regarded as a reference standard used to measure the correlation between bones and pixels in the current video stream to be recognized. Matching can be done by comparing similarity, for example, calculating a similarity metric (such as cosine similarity) between the two coefficient sets to find the closest matching result. Through this matching, the difference between the correlation between bones and pixels in the current video frame and the standard case can be determined. If, during the matching process, the correlation coefficient between a bone point and a pixel grid region is found to be very close to the corresponding correlation coefficient in the standard library (e.g., the difference is less than a first preset value), then an appropriate value will be assigned to the corresponding position in the matrix (e.g., a larger value indicates a strong correlation), which can be set according to the actual use case. Conversely, if the difference is large, a smaller value will be assigned, and the specific value can be set according to the actual use case. In this way, by matching and processing all video frames, a bone-pixel correlation coefficient matrix that can comprehensively reflect the dynamic correlation between bones and pixels in the entire video stream is constructed.

[0068] Furthermore, the determination of the first pixel feature information based on the skeletal pixel correlation coefficient matrix, the preset correlation coefficient threshold, and the second pixel feature information specifically includes:

[0069] The skeletal pixel correlation coefficient matrix is ​​traversed, and correlation coefficients greater than a preset correlation coefficient threshold are selected as the filtering correlation coefficients. The pixel corresponding to the filtering correlation coefficient in the second pixel feature information is determined as the first pixel, and the first pixel feature information is determined based on the selected first pixel.

[0070] In other words, this application filters the correlation coefficients in the bone pixel correlation coefficient matrix by setting a preset correlation coefficient threshold, and finds those correlation coefficients with a high degree of correlation between bones and pixels. Based on the correspondence between the elements in the bone pixel correlation coefficient matrix and the pixels in the second pixel feature information, the first pixel corresponding to the filtered correlation coefficient is determined. Then, based on these filtered first pixels, the feature vector corresponding to the first pixel can be extracted from the second pixel feature information, thereby determining the first pixel feature information.

[0071] The above scheme can more accurately reflect the features of a face, which helps to improve the accuracy and effect of face recognition. In particular, the feature information of the first pixel can play an important role in processing facial feature analysis related to bones and dealing with complex scenes (such as occlusion).

[0072] S102, the server determines the corresponding second three-dimensional skeleton coordinate sequence based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information and the preset adaptive distillation model.

[0073] The second three-dimensional skeletal coordinate sequence is the three-dimensional skeletal coordinate sequence after the occluded face of the object to be identified is completed.

[0074] In this embodiment of the application, the determination of the corresponding second three-dimensional skeleton coordinate sequence based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and the preset adaptive distillation model specifically includes:

[0075] The first 3D skeletal coordinate sequence and the first pixel feature information are input into a pre-set adaptive distillation model to match the initial complete skeletal coordinate sequence in a standard face template library using a pre-trained teacher model. A student model performs convolution processing on the pre-processed first pixel feature information and the first 3D skeletal coordinate sequence to determine the undetermined 3D skeletal coordinate sequence. The student model is trained based on several 3D skeletal coordinate sequence samples of occluded faces. A knowledge distillation loss function is calculated based on the initial complete skeletal coordinate sequence and the undetermined 3D skeletal coordinate sequence. If the knowledge distillation loss function value is less than a preset threshold, the undetermined 3D skeletal coordinate sequence is used as the second 3D skeletal coordinate sequence.

[0076] In other words, this application utilizes a pre-defined adaptive distillation model comprising a teacher model and a student model. First, the pre-trained teacher model, such as a Convolutional Neural Network (CNN), matches the input data with an initial complete skeletal coordinate sequence from a standard face template library. This standard face template library stores various standard, complete face skeletal coordinate sequences, which can serve as reference standards. The teacher model compares the input skeletal coordinate sequence with the sequences in the template library to find the best-matching initial complete skeletal coordinate sequence. The purpose of this step is to provide a more accurate reference framework for subsequent skeletal coordinate sequence generation, using standard skeletal information to guide the model's learning and generation process.

[0077] This application can also preprocess the input first pixel feature information and the first three-dimensional skeleton coordinate sequence (e.g., normalization, feature enhancement, etc.), and then input the preprocessed data into the student model. The student model can perform convolution processing on the input data based on techniques such as convolutional neural networks (CNN). Convolution processing can automatically extract feature information from the data, and through the operation of multiple convolutional layers, the complex relationship between facial bones and pixel features can be learned.

[0078] After processing by the student model, a sequence of undetermined 3D skeletal coordinates is output. This sequence is a possible sequence of 3D skeletal coordinates predicted by the student model based on the input data, but it is not necessarily the final accurate result. Since the student model is trained based on several 3D skeletal coordinate sequence samples of occluded faces, it has a certain ability to process skeletal information under occlusion and can attempt to recover the skeletal coordinates of the occluded parts.

[0079] This application also calculates a knowledge distillation loss function based on the initial complete skeletal coordinate sequence obtained from the teacher model matching and the undetermined 3D skeletal coordinate sequence output by the student model. The knowledge distillation loss function measures the degree of difference between the undetermined sequence output by the student model and the reference sequence provided by the teacher model. The knowledge distillation loss function can use mean squared error, calculating the average of the sum of squares of the differences between corresponding coordinate points in the two sequences to quantify the degree of difference. Simultaneously, this application sets a preset threshold, which is compared with the knowledge distillation loss function value to determine whether the accuracy of the undetermined 3D skeletal coordinate sequence meets the requirements. When the calculated knowledge distillation loss function value is less than the preset threshold, it indicates that the undetermined sequence output by the student model is sufficiently close to the reference sequence of the teacher model. In this case, the undetermined 3D skeletal coordinate sequence can be used as the second 3D skeletal coordinate sequence. The preset threshold can be set by the user according to the actual usage scenario; this application does not impose specific limitations on it.

[0080] The aforementioned second 3D skeletal coordinate sequence further refines and supplements the initial first 3D skeletal coordinate sequence. Especially for occluded facial portions, through model learning and processing, more complete skeletal coordinate information is obtained, providing more accurate skeletal feature data for subsequent tasks such as face recognition. Utilizing a pre-defined adaptive distillation model, combined with the collaboration of the teacher and student models, starting from the first 3D skeletal coordinate sequence and the first pixel feature information, through matching, convolution processing, loss calculation, and judgment, the corresponding second 3D skeletal coordinate sequence is finally determined, improving the accuracy and completeness of the skeletal coordinate information.

[0081] S103, the server calculates the coupling degree value corresponding to each selected pixel in the preset area according to the preset coupling degree calculation formula, the coordinate sequence of each first three-dimensional skeleton and the feature information of each first pixel, and determines the corresponding multimodal feature aggregation amount based on the weighted calculation result of each coupling degree value, the feature information of the first pixel and the coordinate sequence of the second three-dimensional skeleton.

[0082] The coupling degree values ​​mentioned above are used to characterize the degree of dynamic association between bones and pixels.

[0083] In this embodiment of the application, the above-mentioned calculation of the coupling degree value corresponding to each selected pixel in the preset area based on the preset coupling degree calculation formula, the coordinate sequence of each first three-dimensional skeleton, and the feature information of each first pixel specifically includes:

[0084] The variance of the coordinate displacement of each bone point is calculated based on the coordinate sequence of each first 3D skeleton. The reciprocal of the variance of the coordinate displacement of each bone point is used as the temporal stability value of the bone point. The texture entropy of each first pixel point is calculated based on its feature information. The Euclidean distance between each bone point and each selected pixel point is determined based on the coordinate sequence of each first 3D skeleton and the feature information of each first pixel point. The correlation coefficient of the selected pixels is greater than a preset correlation coefficient threshold. The temporal stability value of the bone points, the texture entropy of the pixels, and the Euclidean distance value are input into a preset coupling degree calculation formula to calculate the coupling degree value of the selected pixels.

[0085] In other words, this application uses a first three-dimensional skeleton coordinate sequence and first pixel feature information, along with a preset coupling degree calculation formula, to calculate a coupling degree value that characterizes the dynamic correlation between the skeleton and the pixel. The preset coupling degree calculation formula is as follows:

[0086]

[0087] in, p represents the pixel in the i-th row and j-th column within the preset region at time t. ij (i.e., the selected pixel) and the kth skeletal point b k The coupling degree between them; Represents pixel p at time t ij The corresponding pixel texture entropy; Represents the skeletal point b at time t. k The corresponding temporal stability value of the skeletal point; Represents pixel p at time t ij That is, the coordinates of the selected pixel and the skeleton point b k The Euclidean distance between the coordinates on the two-dimensional plane. Represents pixel p ij Coordinates projected onto a two-dimensional plane. Represents skeletal point b kThe coordinates projected onto the two-dimensional plane; ∈ is a preset positive value used to avoid the denominator being zero and to ensure the stability of the formula; This is the time decay term, where t is the current time. ref τ is a preset reference time, which can be set by the user according to the actual use scenario, and is not specifically limited here; τ is a preset time decay coefficient, which can be set by the user according to the actual use scenario, and is not specifically limited here. It is used to control the decay rate of the coupling value over time.

[0088] The above-mentioned skeletal point temporal stability values The calculation formula is as follows:

[0089] First, calculate the skeletal point b. k The variance of the skeletal point coordinate displacement at n consecutive time points: Then through Calculate the temporal stability value of the skeletal points. (σ) x ,σ y ,σ z ) represents the skeletal point b. k coordinates (x) k ,y k ,z k The variance of the skeletal point coordinate displacement over n consecutive time points. Let be the average coordinates over n consecutive time points.

[0090] The above pixel texture entropy The calculation formula is as follows:

[0091]

[0092] in, Let (i, j) be the gray value of pixel (i, j) at time t. The probability value is g, which represents all possible grayscale values ​​that can be traversed. It can be set according to the actual use case and is not specifically limited here.

[0093] In one embodiment of this application, the determination of the corresponding multimodal feature aggregation amount based on the weighted calculation results of each coupling degree value, the feature information of the first pixel point, and the second three-dimensional skeleton coordinate sequence specifically includes:

[0094] Based on the feature information of each first pixel and the preset occlusion ratio prediction model, the facial occlusion ratio of the object to be identified is determined, and pixel feature weights and skeletal feature weights are determined based on the facial occlusion ratio. The sum of the pixel feature weights and skeletal feature weights is 1. The feature information of each selected pixel in the first pixel feature information is vector-encoded to obtain the first feature vector, and the feature attributes corresponding to each skeletal point in the second three-dimensional skeletal coordinate sequence are vector-encoded to determine the second feature vector. The feature attributes corresponding to the skeletal points include at least position encoding, motion velocity, and pose. Based on the pixel feature weights and skeletal feature weights, the weighted sum of the first feature vector and the second feature vector of the kth skeletal point is calculated, and the weighted sum is used as the kth first component. k is the total number of selected pixels, and k is a natural number. The product of the coupling degree value of the kth first component and the kth skeletal point is calculated as the kth second component, and the sum of the k second components is calculated to obtain the multimodal feature aggregation amount of the corresponding selected pixel.

[0095] In other words, for each frame of an RGB image, this application can use a pre-trained lightweight preset occlusion ratio prediction model, which can be a neural network model, machine learning model, etc., to identify the feature information of the first pixel and the facial occlusion ratio of the object to be identified. Simultaneously, the server pre-stores the correspondence between different facial occlusion ratios and different feature weight groups. Different feature weight groups contain different pixel feature weights and skeletal feature weights. The correspondence can be set according to the actual use scenario and is not specifically limited here. Among them, the skeletal feature weight and the facial occlusion ratio are inversely proportional; that is, the more the face is occluded, the larger the skeletal feature weight. For example, if the facial occlusion ratio is 0.6, the skeletal feature weight might be set to 0.6 and the pixel feature weight to 0.4.

[0096] Subsequently, this application performs vector encoding on the feature information of each selected pixel in the first pixel feature information, that is, maps the pixel features to a high-dimensional vector space so that each vector can accurately represent the feature information of the pixel; at the same time, for each bone point in the second three-dimensional skeleton coordinate sequence, its corresponding feature attributes (including at least position encoding, motion velocity, and pose) are vector encoded. This yields the first feature vector and the second feature vector, and based on the pixel feature weights and skeleton feature weights obtained above, calculates the weighted sum c of the first feature vector and the second feature vector of the k-th bone point. k Then, based on the coupling degree value mentioned above, the second component c is obtained. k ×C k C k C represents the coupling degree value of the k-th skeletal point. k Is with c k At the corresponding time in the same frame, all C's of the same bone point k ij,kThe server calculates the sum of the second components of the k skeletal points; then, it calculates the sum of the second components of the k skeletal points, and subsequently obtains the multimodal feature aggregation value of the k-th selected pixel (i, j). The formula for calculating the multimodal feature aggregation value is as follows:

[0097] The above scheme comprehensively considers pixel features, skeletal features, and their coupling relationships, effectively aggregating multimodal information to obtain more representative feature quantities, providing richer and more accurate information for subsequent tasks such as face recognition.

[0098] S104, the server determines the skeletal pixel feature vector based on the aggregated amount of each multimodal feature, the second three-dimensional skeletal coordinate sequence and the preset attention mechanism, and determines the face recognition result of the object to be recognized based on the skeletal pixel feature vector and the preset face recognition database.

[0099] In this embodiment of the application, the determination of the bone pixel feature vector based on the aggregation of various multimodal features, the second three-dimensional bone coordinate sequence, and the preset attention mechanism specifically includes:

[0100] The aggregated multimodal features are concatenated with the second 3D skeletal coordinate sequence to obtain a concatenated vector. This concatenated vector is then input into a predefined attention mechanism to map the query vector, key vector, and value vector, and the output value of each attention head is calculated. Finally, the output values ​​of each attention mechanism are concatenated and linearly transformed to obtain the skeletal pixel feature vector.

[0101] In other words, this application can also concatenate the aggregated multimodal features with the second three-dimensional skeletal coordinate sequence, that is, connect them in a certain order to form a new vector. If the aggregated multimodal features are an m-dimensional vector and the second three-dimensional skeletal coordinate sequence is represented as an n-dimensional vector, then the concatenated vector is an (m+n)-dimensional vector. Then, the input to the preset attention mechanism undergoes a linear transformation, mapping it to a query vector Q, a key vector K, and a value vector V. Multi-head attention works in parallel through this preset attention mechanism to calculate the output values ​​of each attention mechanism. This preset attention mechanism can be pre-trained using several concatenated vectors. Then, the server concatenates the output values ​​to obtain an output value concatenated vector and further linearly transforms it to obtain the skeletal pixel feature vector. This further linear transformation can be achieved by multiplying the preset weight matrix with the output value concatenated vector and adding a bias term. The preset weight matrix and bias term can be set according to the actual use case and are not specifically limited here.

[0102] Furthermore, based on the skeletal pixel feature vector and a pre-set face recognition database, the face recognition result of the object to be recognized is determined, specifically including:

[0103] Based on the preset Bayesian decision rule, the posterior probability between the skeletal pixel feature vector and each identity entry information in the preset face recognition database is calculated. The maximum posterior probability value among all posterior probabilities is determined, and the identity identifier corresponding to the maximum posterior probability value is used as the face recognition result of the object to be identified.

[0104] In other words, the server can be pre-configured with a preset face recognition database, which can store the identity information of pre-entered objects. This application uses Bayesian decision rules, taking the skeletal pixel feature vector as the observed feature vector and the identity information as the sample category. It assumes that the feature vector follows a preset probability distribution, such as a Gaussian distribution. This probability distribution can be selected by the user according to the actual scenario, and is not specifically limited here. Then, the class conditional probability density is calculated based on the feature vector data of the identity information. Next, the prior probability is obtained according to the quantity ratio of identity information in the preset face recognition database. Subsequently, the posterior probability corresponding to the skeletal pixel feature vector and different identity information is calculated according to the Bayesian formula. By finding the maximum value among the various posterior probabilities, the identity identifier corresponding to the corresponding identity information can be determined, and it is used as the face recognition result of the object to be identified.

[0105] The above-mentioned method adopts a preset Bayesian decision rule. By calculating the posterior probability of each identity information entered in the preset face recognition database and the skeletal pixel feature vector, it provides a scientific basis for recognition decision from a probabilistic perspective. The identity identifier corresponding to the maximum posterior probability value is used as the recognition result, which enhances the reliability and scientific nature of the recognition result.

[0106] The above scheme, through a pre-set adaptive distillation model and the collaboration of teacher and student models, completes the second three-dimensional skeletal coordinate sequence of the occluded face based on the first three-dimensional skeletal coordinate sequence and the feature information of the first pixel. This provides complete skeletal feature information for accurate recognition, effectively overcoming the interference of occlusion on recognition, improving recognition accuracy in complex occlusion scenarios, and handling different degrees of occlusion and scenarios, thus enhancing the generalization ability of face recognition. Simultaneously, the coupling degree value is calculated to represent the dynamic relationship between bones and pixels, fully exploring the intrinsic connections between multimodal data to calculate the multimodal feature aggregation, making feature expression more comprehensive and enhancing recognition accuracy. This eliminates the need for manual intervention, reducing labor costs. Therefore, it solves the problems of low accuracy, poor stability, weak generalization ability, and high labor costs in face recognition under complex occlusion scenarios.

[0107] Figure 2 A schematic diagram of the structure of a face recognition system based on a skeletal calibration algorithm provided in this application embodiment is shown below. Figure 2 As shown, the face recognition system 200 based on the skeletal calibration algorithm includes:

[0108] The first determining module 201 is used to determine the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be identified within a preset time period based on the face recognition video stream from a preset face recognition device group. The second determining module 202 is used to determine the corresponding second three-dimensional skeleton coordinate sequence based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and a preset adaptive distillation model. The second three-dimensional skeleton coordinate sequence is the three-dimensional skeleton coordinate sequence after completing the occluded face of the object to be identified. The calculation module 203 is used to calculate the coupling degree value corresponding to each selected pixel within a preset area based on a preset coupling degree calculation formula, the first three-dimensional skeleton coordinate sequence, and the feature information of each first pixel. The corresponding multimodal feature aggregation amount is determined based on the weighted calculation result of each coupling degree value, the first pixel feature information, and the second three-dimensional skeleton coordinate sequence. The coupling degree value is used to characterize the dynamic correlation between the skeleton and the pixel. The third determining module 204 is used to determine the skeleton pixel feature vector based on each multimodal feature aggregation amount, the second three-dimensional skeleton coordinate sequence, and a preset attention mechanism. Based on the skeleton pixel feature vector and a preset face recognition database, the face recognition result of the object to be identified is then determined.

[0109] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0110] The systems and methods provided in this application are one-to-one correspondences. Therefore, the system also has similar beneficial technical effects as its corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system will not be repeated here.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A face recognition method based on a skeletal calibration algorithm, characterized in that, The method includes: Based on a face recognition video stream from a preset face recognition device group, the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be recognized within a preset time period are determined. Specifically, this includes: dividing each facial RGB image into a grid according to a preset grid to determine the feature vector corresponding to each facial RGB image as the feature information of the second pixel; determining the corresponding skeleton pixel correlation coefficient matrix according to each first three-dimensional skeleton coordinate sequence and each second pixel feature information; traversing the skeleton pixel correlation coefficient matrix to filter correlation coefficients that are greater than a preset correlation coefficient threshold; determining the pixel corresponding to the filtered correlation coefficient in the second pixel feature information as the first pixel, and determining the first pixel feature information based on the filtered first pixel; Based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and the preset adaptive distillation model, a corresponding second three-dimensional skeleton coordinate sequence is determined; wherein, the second three-dimensional skeleton coordinate sequence is the three-dimensional skeleton coordinate sequence after completing the occluded face of the object to be identified; Based on the preset coupling degree calculation formula, the coordinate sequences of each first three-dimensional skeleton, and the feature information of each first pixel, the coupling degree value corresponding to each selected pixel in the preset region is calculated. The corresponding multimodal feature aggregation amount is determined based on the weighted calculation result of each coupling degree value, the feature information of the first pixel, and the coordinate sequence of the second three-dimensional skeleton. The coupling degree value is used to characterize the dynamic correlation between the skeleton and the pixel. The correlation coefficient of the selected pixel is greater than the preset correlation coefficient threshold. Based on the aggregated multimodal features, the second three-dimensional skeletal coordinate sequence, and the preset attention mechanism, the skeletal pixel feature vector is determined, and the face recognition result of the object to be recognized is determined according to the skeletal pixel feature vector and the preset face recognition database.

2. The face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, Based on the face recognition video stream from a preset group of face recognition devices, the system determines the first three-dimensional skeletal coordinate sequence and the feature information of each first pixel of the object to be recognized within a preset time period. Specifically, this also includes: The face recognition video stream from the preset face recognition device group is input into a pre-trained skeletal calibration model to determine the coordinates of several approximate skeletal points in the face recognition video stream corresponding to the object to be recognized based on the model output results. Map the coordinates of each of the approximate bone points to a preset depth map coordinate system, and add the mapped bone coordinate points to each of the first three-dimensional bone coordinate sequences in chronological order. Determine the RGB images of each face in the face recognition video stream; wherein each frame of the face recognition video stream contains an RGB image and a depth image.

3. The face recognition method based on a skeletal calibration algorithm according to claim 2, characterized in that, Based on the coordinate sequences of each of the first three-dimensional skeletons and the feature information of each of the second pixels, the corresponding skeleton pixel correlation coefficient matrix is ​​determined, specifically including: Based on the first three-dimensional skeleton coordinate sequence and the second pixel feature information corresponding to the same frame, the dynamic skeleton pixel correlation coefficient is determined, and a set of dynamic skeleton pixel correlation coefficients corresponding to each video frame is constructed; the dynamic skeleton pixel correlation coefficient is calculated based on the distance between the skeleton mapping coordinate point and the corresponding grid center; The dynamic skeleton pixel correlation coefficient set is matched with a preset standard correlation coefficient library to determine the skeleton pixel correlation coefficient matrix based on the matching result; wherein, the preset standard correlation coefficient set is constructed based on the unoccluded face recognition video stream of the pre-recorded object.

4. The face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, Based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and the preset adaptive distillation model, a corresponding second three-dimensional skeleton coordinate sequence is determined, specifically including: The first three-dimensional skeletal coordinate sequence and the first pixel feature information are input into the preset adaptive distillation model to match the initial complete skeletal coordinate sequence in the standard face template library through the pre-trained teacher model. The student model is used to perform convolution processing on the preprocessed feature information of the first pixel and the first three-dimensional skeleton coordinate sequence to determine the three-dimensional skeleton coordinate sequence to be determined; the student model is trained based on several three-dimensional skeleton coordinate sequence samples of occluded faces. The knowledge distillation loss function value is calculated based on the initial complete skeleton coordinate sequence and the undetermined three-dimensional skeleton coordinate sequence. If the knowledge distillation loss function value is less than a preset threshold, the undetermined three-dimensional skeleton coordinate sequence is used as the second three-dimensional skeleton coordinate sequence.

5. A face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, Based on the preset coupling degree calculation formula, the coordinate sequences of each of the first three-dimensional skeletons, and the feature information of each of the first pixels, the coupling degree value corresponding to each selected pixel within the preset region is calculated, specifically including: The displacement variance of each bone point is calculated based on each of the first three-dimensional bone coordinate sequences, and the reciprocal of the displacement variance of the bone point is used as the temporal stability value of the bone point. The texture entropy of the pixel point is calculated based on the feature information of each of the first pixel points. Based on the coordinate sequences of each of the first three-dimensional skeletons and the feature information of the first pixel, the Euclidean distance between each skeleton and each of the selected pixels is determined. The temporal stability value of the skeletal point, the texture entropy of the pixel point, and the Euclidean distance value are input into the preset coupling degree calculation formula to calculate the coupling degree value of the selected pixel point.

6. The face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, The corresponding multimodal feature aggregation amount is determined based on the weighted calculation results of each coupling degree value, the first pixel feature information, and the second three-dimensional skeleton coordinate sequence, specifically including: Based on the feature information of each first pixel and the preset occlusion ratio prediction model, the facial occlusion ratio of the object to be identified is determined, and the pixel feature weight and the skeletal feature weight are determined based on the facial occlusion ratio; the sum of the pixel feature weight and the skeletal feature weight is 1. The feature information of each selected pixel in the first pixel feature information is vector-encoded to obtain a first feature vector, and the feature attributes corresponding to each bone point in the second three-dimensional bone coordinate sequence are vector-encoded to determine a second feature vector; the feature attributes corresponding to the bone point include at least position encoding, motion speed and posture. Based on the pixel feature weights and the bone feature weights, calculate the first feature vector and the second feature vector. The weighted sum of the second feature vectors of the _n bone points is used as the _n _ ... The first component; It is a natural number; Calculate the first The first component and the first The product of the coupling values ​​of the 1st skeletal point is used as the 1st... The second component is calculated, and the sum of each second component is obtained to obtain the multimodal feature aggregation amount of the selected pixel.

7. A face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, Based on the aggregated multimodal features, the second three-dimensional skeleton coordinate sequence, and the preset attention mechanism, the skeleton pixel feature vector is determined, specifically including: The aggregated multimodal features are concatenated with the second three-dimensional skeleton coordinate sequence to obtain a concatenated vector; The concatenated vector is input into the preset attention mechanism to map the query vector, key vector, and value vector, and the attention mechanism output value corresponding to each attention head is calculated. The output values ​​of each attention mechanism are concatenated and linearly transformed to obtain the skeletal pixel feature vector.

8. A face recognition method based on a skeletal calibration algorithm according to claim 1, characterized in that, Based on the skeletal pixel feature vector and a preset face recognition database, the face recognition result of the object to be recognized is determined, specifically including: According to the preset Bayesian decision rule, the posterior probability between the skeletal pixel feature vector and each identity entry information in the preset face recognition database is calculated respectively. The maximum posterior probability value among the posterior probabilities is determined, and the identity identifier corresponding to the maximum posterior probability value is used as the face recognition result of the object to be identified.

9. A face recognition system based on a skeletal calibration algorithm, characterized in that, The system is capable of executing the face recognition method based on a skeleton calibration algorithm as described in any one of claims 1-8; the system includes: The first determining module is used to determine the first three-dimensional skeleton coordinate sequence and the feature information of each first pixel of the object to be identified within a preset time period based on the face recognition video stream from the preset face recognition device group. The second determining module is used to determine a corresponding second three-dimensional skeleton coordinate sequence based on the first three-dimensional skeleton coordinate sequence, the first pixel feature information, and a preset adaptive distillation model; wherein, the second three-dimensional skeleton coordinate sequence is a three-dimensional skeleton coordinate sequence after completing the occluded face of the object to be identified. The calculation module is used to calculate the coupling degree value corresponding to each selected pixel in a preset area according to the preset coupling degree calculation formula, the coordinate sequence of each first three-dimensional skeleton and the feature information of each first pixel, and to determine the corresponding multimodal feature aggregation amount based on the weighted calculation result of each coupling degree value, the feature information of the first pixel and the coordinate sequence of the second three-dimensional skeleton; the coupling degree value is used to characterize the dynamic correlation between the skeleton and the pixel. The third determining module is used to determine the skeletal pixel feature vector based on the aggregated amount of each multimodal feature, the second three-dimensional skeletal coordinate sequence and the preset attention mechanism, and to determine the face recognition result of the object to be recognized based on the skeletal pixel feature vector and the preset face recognition database.

Images