Facial feature recognition method and device based on semi-supervised pre-training

Abstract

The invention provides a facial feature recognition method and device based on semi-supervised pre-training, and relates to the technical field of facial recognition, and the method comprises the steps: generating a face segmentation data set based on a face image set in a target scene; dividing the face segmentation data set into a supervised sample data set and a non-supervised sample data set for performing semi-supervised pre-training on an identification backbone network in the facial feature identification network, freezing parameters of the identification backbone network, and utilizing a pseudo tag of the non-supervised sample data set and the supervised sample data set to identify the facial feature identification network; and training the multi-subclass prediction head in the facial feature recognition network to obtain a target facial feature recognition network, the target facial feature recognition network being used for predicting facial features of the target object. According to the method, the problems of high data labeling cost and low accuracy of a trained model of face feature recognition in a medical scene can be solved.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to a facial feature recognition method and apparatus based on semi-supervised pre-training. Background Technology

[0002] The application of facial feature recognition in the current medical field is receiving increasing attention. Human facial features can provide crucial information about specific diseases, such as a crooked mouth, facial stiffness, and drooling, improving doctors' diagnostic efficiency and accuracy. They can also monitor patients' sensory and emotional states, such as pain or anxiety, enhancing humanistic care during medical diagnosis and rehabilitation. In recent years, neural network-based facial feature recognition algorithms have made significant progress, with model prediction accuracy greatly improved. However, practical applications in the medical field often require large amounts of data labeled with doctors' professional knowledge. This results in extremely high labeling costs to achieve target performance; conversely, insufficient labeled data leads to a lack of model generalization ability, resulting in inadequate model performance. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a facial feature recognition method and device based on semi-supervised pre-training, which can overcome the problems of high cost of labeled data and low accuracy of trained models in medical scenarios.

[0004] In a first aspect, the present invention provides a facial feature recognition method based on semi-supervised pre-training, comprising: Generate a face segmentation dataset based on a set of face images from the target scene; The face segmentation dataset is divided into supervised sample datasets and unsupervised sample datasets; Using supervised and unsupervised sample datasets, the recognition backbone network in the facial feature recognition network is semi-supervised pre-trained. After pre-training, the parameters of the recognition backbone network are frozen. Then, using supervised sample datasets, unsupervised sample datasets, and pseudo-labels generated for the unsupervised sample datasets, the multi-subclass prediction head in the facial feature recognition network is trained to obtain the target facial feature recognition network, which is used to predict the facial features of the target object.

[0005] In one implementation, a face segmentation dataset is generated based on a set of face images in the target scene, including: Strong enhancement and weak enhancement processing are performed on the first face image with face detection boxes in the face image set to obtain strong enhancement face image and weak enhancement face image respectively; Based on weakly enhanced face images, determine the segmentation results of strongly enhanced face instances corresponding to strongly enhanced face images; The face instance segmenter is trained using strongly enhanced face images and their corresponding strongly enhanced face instance segmentation results; Using the trained face instance segmenter, face segmentation is performed on the first face image with face detection boxes labeled and the second face image without face detection boxes labeled in the face image set, resulting in a generated face segmentation dataset.

[0006] In one implementation, determining the segmentation result of a strongly enhanced face instance corresponding to a strongly enhanced face image based on a weakly enhanced face image includes: Weak enhancement processing is applied to the face detection bounding boxes annotated in the first face image, and the weakly enhanced face detection bounding boxes are then horizontalized to obtain weakly enhanced face detection bounding boxes. Using an image segmentation model based on the SAM architecture, weakly enhanced face instance segmentation results are generated based on weakly enhanced face images and weakly enhanced face detection boxes. Determine the affine transformation matrix between the strongly enhanced face image and the weakly enhanced face image; Based on the affine transformation matrix and the weakly enhanced face instance segmentation results, the strongly enhanced face instance segmentation results corresponding to the strongly enhanced face images are determined.

[0007] In one implementation, semi-supervised pre-training is performed on the recognition backbone network of the facial feature recognition network using supervised sample datasets and unsupervised sample datasets, including: Perform the following operations in the current iteration batch: The supervised sample dataset and the unsupervised sample dataset are input into the target neural network, which includes an improved Query network and an improved Key network, to obtain the first facial feature vector and the first feature type prediction probability output by the Query network, and the first facial feature vector and the first feature type prediction probability output by the Key network. Based on the first facial feature vector and the first feature type prediction probability output by the Key network for the unsupervised sample dataset, the first pseudo-label is assigned to the unsupervised sample dataset. Based on the first facial feature vector output by the Key network for supervised sample data, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined. Based on the first pseudo-label labeled in the unsupervised sample dataset, the true label labeled in the supervised sample dataset, and the quality value corresponding to the first pseudo-label, a target loss value is determined. The network parameters of the Query network are updated using the target loss value, and the network parameters of the Key network are updated using the network parameters of the Query network. The true label is the facial feature type to which the supervised sample belongs. When the current iteration batch meets the preset stopping conditions, the Query network will be used as the recognition backbone network in the facial feature recognition network.

[0008] In one implementation, both the Query network and the Key network include backbone units, a projection head, and a prediction head. The backbone network includes the backbone units and projection head of the Query network. Supervised and unsupervised sample datasets are input into a target neural network, which includes an improved Query network and an improved Key network, to obtain a first facial feature vector and a first feature type prediction probability output by the Query network, and a first facial feature vector and a first feature type prediction probability output by the Key network, including: For each sample in the supervised and unsupervised sample datasets of the current iteration, perform the following operation: After performing different image enhancement processes on the sample, the images were input into the Query network and the Key network, respectively. The initial feature vector corresponding to the sample after image enhancement is extracted through the backbone network in the Query network. The first facial feature vector is generated based on the initial feature vector through the projection head in the Query network. The first feature type prediction probability is generated based on the initial feature vector through the prediction head in the Query network. Furthermore, the initial feature vector corresponding to another image-enhanced sample is extracted through the backbone network in the Key network, the first facial feature vector is generated based on the initial feature vector through the projection head in the Key network, and the first feature type prediction probability is generated based on the initial feature vector through the prediction head in the Key network.

[0009] In one implementation, a pre-built memory stores historical facial feature vectors and historical feature type prediction probabilities output by the Key Network for supervised and unsupervised sample datasets under historical iteration batches; based on the first facial feature vector and first feature type prediction probability output by the Key Network for the unsupervised sample dataset, a first pseudo-label is assigned to the unsupervised sample dataset, including: For each unsupervised sample in the unsupervised sample dataset of the current iteration, perform the following operation: The first facial feature vector output by the Key network for this unsupervised sample is denoted as the first feature vector. Determine the first inner product similarity between the first feature vector and the historical facial feature vectors stored in the memory bank, so as to filter out multiple historical facial feature vectors that match the first feature vector from the memory bank; Based on the first inner product similarity, the predicted probability of the historical feature type corresponding to multiple historical facial feature vectors matched by the first feature vector is weighted and averaged to obtain the predicted probability of the target feature type of the unsupervised sample belonging to each facial feature type. If both the target feature type prediction probability and the first feature type prediction probability output by the Key network for the unsupervised sample are greater than a preset threshold, the facial feature type corresponding to the highest target feature type prediction probability will be used as the first pseudo-label of the unsupervised sample.

[0010] In one implementation, based on the first facial feature vector output by the Key network for supervised sample data, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined, including: For each supervised sample in the supervised sample dataset of the current iteration, perform the following operation: The first facial feature vector output by the Key network for this supervised sample is denoted as the second feature vector; Determine the second inner product similarity between the second feature vector and the historical facial feature vectors stored in the memory bank, in order to filter out multiple historical facial feature vectors that match the second feature vector from the memory bank; Based on the second inner product similarity, the predicted probability of the historical feature type corresponding to multiple historical facial feature vectors matched by the second feature vector is weighted and averaged to obtain the predicted probability of the target feature type of the supervised sample belonging to each facial feature type. Based on the predicted probability of the target feature type corresponding to each supervised sample in the supervised sample dataset in this round of iteration, and the average value of the KL divergence between the labeled facial feature types, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset in this round of iteration is determined.

[0011] In one implementation, the target loss value is determined based on the first pseudo-label labeled in the unsupervised sample dataset, the true label labeled in the supervised sample dataset, and the quality value corresponding to the first pseudo-label, including: The semi-supervised contrastive learning loss value is determined based on the first pseudo-label labeled in the unsupervised sample dataset and the true label labeled in the supervised sample dataset; and the regression loss value is determined based on the true label labeled in the supervised sample dataset and the predicted probability of the corresponding target feature type. The target loss value is obtained by fusing the regression loss value and the semi-supervised contrastive learning loss value with the quality value corresponding to the first pseudo-label of the unsupervised sample dataset.

[0012] In one implementation, determining the semi-supervised contrastive learning loss value based on the first pseudo-labels annotated in the unsupervised sample dataset and the true labels annotated in the supervised sample dataset includes: Construct a list of first feature vectors based on the first facial feature vector output by the Query network; A second feature vector list is constructed based on the first facial feature vector output by the Key network and the historical facial feature vectors stored in the memory bank. For a target sample labeled with a true label or a first pseudo label in the first feature vector list, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is labeled with the same true label or first pseudo label as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. For a target sample in the first feature vector that is not labeled with a true label or a first pseudo label, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is the same as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. The semi-supervised contrastive learning loss value is determined based on the positive and negative matching sets.

[0013] In one implementation, the multi-subclass prediction head is a facial feature template vector library containing multiple templates for each facial feature type, including normal feature classes and one or more abnormal feature classes. The multi-subclass prediction head in the facial feature recognition network is trained using supervised sample datasets, unsupervised sample datasets, and pseudo-labels generated for the unsupervised sample datasets, including: By identifying the backbone network, the second facial feature vector corresponding to the unsupervised sample data is determined, and the prediction head in the Query network is used to predict the second feature type prediction probability of the unsupervised sample dataset belonging to each facial feature type. Based on the second feature type prediction probability, the unsupervised sample dataset is assigned a second pseudo label. Determine the cosine similarity between the second facial feature vector and the template corresponding to each abnormal feature class, and take the largest cosine similarity as the first cosine similarity between the second facial feature vector and each abnormal feature class; Determine the cosine similarity between the second facial feature vector and the template corresponding to the normal feature class, and take the largest cosine similarity as the second cosine similarity between the second facial feature vector and the normal feature class; The first cosine similarity and the second cosine similarity are concatenated to obtain the target cosine similarity. Based on the second pseudo-label corresponding to the unsupervised sample dataset and the true label labeled in the supervised sample dataset, the occurrence probability of each facial feature type is calculated, and the prediction head loss value is determined by combining the target cosine similarity. The prediction head loss value is then used to update the network parameters of the multi-subclass prediction head.

[0014] In one implementation, the method further includes: Obtain the face image to be processed; The face segmentation result corresponding to the face image to be processed is extracted using a face instance segmenter. The third facial feature vector corresponding to the face segmentation result is extracted through the recognition backbone network in the facial feature recognition network. Then, the third facial feature vector corresponding to the face segmentation result is compared with the facial feature template vector library through the multi-subclass prediction head in the facial feature recognition network to obtain the facial features corresponding to the face image to be processed.

[0015] Secondly, the present invention also provides a facial feature recognition device based on semi-supervised pre-training, comprising: The face segmentation module is used to generate a face segmentation dataset based on a set of face images in a target scene. The dataset partitioning module is used to divide the face segmentation dataset into a supervised sample dataset and an unsupervised sample dataset. The supervised sample dataset contains supervised samples labeled with true labels, and the true labels are the facial feature types to which the supervised samples belong. The feature recognition network training module is used to perform semi-supervised pre-training on the recognition backbone network of the facial feature recognition network using supervised and unsupervised sample datasets. After pre-training, the parameters of the recognition backbone network are frozen. Then, the multi-subclass prediction head of the facial feature recognition network is trained using supervised sample datasets, unsupervised sample datasets, and pseudo-labels generated for the unsupervised sample datasets to obtain the target facial feature recognition network, which is used to predict the facial features of the target object.

[0016] This invention provides a facial feature recognition method and apparatus based on semi-supervised pre-training. First, a face segmentation dataset is generated based on a set of face images from a target scene. Then, the face segmentation dataset is divided into a supervised sample dataset and an unsupervised sample dataset. Finally, using the supervised and unsupervised sample datasets, a semi-supervised pre-training process is performed on the recognition backbone network of the facial feature recognition network. After pre-training, the parameters of the recognition backbone network are frozen, and the multi-class prediction head in the facial feature recognition network is trained using the supervised sample dataset, the unsupervised sample dataset, and pseudo-labels generated for the unsupervised sample dataset to obtain the target facial feature recognition network. The target facial feature recognition network is used to predict the facial features of the target object. This method uses semi-supervised training of the recognition backbone network and, after freezing the recognition backbone network, uses both real labels from supervised data and pseudo-labels from unsupervised data to jointly train the multi-class prediction head. This fully utilizes the information in the unsupervised sample dataset, improving model training performance with the same amount of annotations, thereby overcoming the problems of high cost of labeled data and low model accuracy in medical scenarios for facial feature recognition.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a facial feature recognition method based on semi-supervised pre-training provided in an embodiment of the present invention; Figure 2 A technical framework diagram of a facial feature recognition method based on semi-supervised pre-training provided for an embodiment of the present invention; Figure 3 A schematic diagram of a semi-supervised training model backbone network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a disease feature vector template provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a facial feature recognition device based on semi-supervised pre-training, provided as an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, in practical applications in the medical field, a large amount of data labeled with doctors' professional knowledge is often required, which makes the labeling cost particularly high in order to achieve the target performance. On the other hand, insufficient labeling can lead to a lack of model generalization ability and insufficient model performance. Based on this, this invention provides a facial feature recognition method and device based on semi-supervised pre-training, which can overcome the problems of high cost of labeled data and low accuracy of trained models in medical scenarios.

[0023] To facilitate understanding of this embodiment, a facial feature recognition method based on semi-supervised pre-training disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 1 The diagram shows a flowchart of a facial feature recognition method based on semi-supervised pre-training. The method mainly includes the following steps S102 to S106: Step S102: Generate a face segmentation dataset based on the face image set in the target scene.

[0024] In one example, cameras can be deployed in the target scene to collect multiple face images in the target scene, forming a face image dataset. A face instance segmenter is trained using a first subset of face images with labeled face detection boxes in the face image dataset. The trained face instance segmenter is then used to segment faces in the first subset of face images and the second subset of face images without labeled face detection boxes, resulting in a face segmentation dataset.

[0025] The face instance segmenter can employ the YoloV11-seg model. In one implementation, a subset of face images in the face image dataset can be labeled based on available manpower, resulting in a first subset of face images labeled with face detection boxes. Face images without labeled face detection boxes constitute a second subset of face images. Based on this, the YoloV11-seg model is first trained using the first subset of face images, combined with Segment-Anything-Model (SAM) distillation. The trained YoloV11-seg model serves as the face instance segmenter, inputting both the first and second subsets of face images to obtain a high-precision face segmentation dataset.

[0026] Step S104: Divide the face segmentation dataset into a supervised sample dataset and an unsupervised sample dataset.

[0027] As is understandable, supervised samples are face segmentation data with true labels, while unsupervised samples are face segmentation data without labels. Supervised samples are labeled with true labels indicating their respective facial feature types. These facial feature types can be normal or abnormal. Normal features can be understood as the facial features of the target object under normal conditions, while abnormal features can be the facial features of the target object under disease conditions, such as a crooked mouth, facial stiffness, or drooling. In one implementation, the face segmentation dataset can be labeled according to available manpower, resulting in a mixed dataset containing partially labeled data (i.e., supervised samples) and other unlabeled data (i.e., unsupervised samples).

[0028] Step S106: Using supervised and unsupervised sample datasets, semi-supervised pre-training is performed on the recognition backbone network in the facial feature recognition network. After pre-training, the parameters of the recognition backbone network are frozen. Then, using supervised sample datasets, unsupervised sample datasets, and pseudo-labels generated for the unsupervised sample datasets, the multi-subclass prediction head in the facial feature recognition network is trained to obtain the target facial feature recognition network.

[0029] The facial feature recognition network comprises a recognition backbone network and a multi-subclass prediction head. The recognition backbone network is obtained by training the target neural network using a semi-supervised training method. The target neural network can be a MoCoV series network, including MoCoV2 and MoCoV3. Taking MoCoV2 as an example, this network includes an improved Query network and an improved Key network. Compared to the traditional MoCoV2, this embodiment introduces prediction heads into both the Query and Key networks. That is, both the improved Query and Key networks include backbone units, projection heads, and prediction heads. Specifically, the recognition backbone network refers to the backbone units and projection heads in the Query network. The multi-subclass prediction head is a facial feature template vector library, which contains multiple templates for each facial feature type. The target facial feature recognition network, i.e., the trained facial feature recognition network, is used to predict the facial features of the target object. In one implementation, the unsupervised contrastive learning algorithm MoCoV2 is modified to adapt for semi-supervised learning. Then, the modified algorithm is used to train a recognition backbone network on the training set. Next, the network parameters of the recognition backbone network are frozen, and a multi-subclass prediction head is trained using labeled training set data and pseudo-labels to obtain multiple templates for each facial feature type and the absence of a corresponding feature. These templates are then stored as a template vector library, completing the entire model training process. The pseudo-labels can be either the first pseudo-label with high confidence from the semi-supervised learning process or a second pseudo-label assigned to the unsupervised sample data using the trained prediction head.

[0030] The facial feature recognition method based on semi-supervised pre-training provided in this invention uses semi-supervised training to identify the backbone network, and after freezing the backbone network, it uses supervised data labels and unsupervised data pseudo-labels to jointly train the multi-subclass prediction head. This fully exploits the information in the unsupervised sample dataset, improves the model training performance with the same number of labels, and overcomes the problems of high cost of labeled data and low accuracy of trained models in medical scenarios for facial feature recognition.

[0031] In one implementation, see Figure 2 The diagram shows a technical framework for a facial feature recognition method based on semi-supervised pre-training, including a model training part and a data inference part.

[0032] The model training process includes: S1, deploying cameras and collecting images; S2, labeling and training the face instance segmentation model YoloV11-seg using Segment-Anything-Model (SAM) distillation; S3, using the face instance segmentation model YoloV11-seg to obtain the face segmentation dataset for all collected data; S4, labeling facial features in a portion of the face segmentation dataset; S5, modifying the unsupervised contrastive learning algorithm MoCoV2 to adapt to semi-supervised learning; S6, freezing the model's recognition backbone network, using the labeled data from the training set, and inputting the high-confidence pseudo-labels from the semi-supervised learning process into the model to train a multi-subclass prediction head to obtain multiple templates for each facial feature and no-feature counterparts, which are then stored as a facial feature template vector library.

[0033] The data inference part includes: S8, obtaining face segmentation using the trained face instance segmenter; S9, inputting the face segmentation into the recognition backbone network in the facial feature recognition network to extract facial feature vectors; S10, comparing the facial feature vectors with the facial feature template vector library, and when the maximum similarity is greater than the threshold, the match is considered successful and the predicted facial features are output.

[0034] To facilitate understanding, the embodiments of the present invention will first explain the model training part.

[0035] S1, Deploy cameras and collect images.

[0036] In one implementation, a camera is installed in the target scene, and an angle and position are selected that can clearly see the target face without causing psychological discomfort to the subject being tested. The video captured by the camera is then saved as an image (i.e., a set of face images).

[0037] S2, annotated and trained using Segment-Anything-Model (SAM) distillation, trains the face instance segmentation model YOLOv11-seg. Specifically, it includes: S2.1, Based on the specific manpower situation, select a portion of the face image set and use LabelMe to annotate the face detection boxes. Specifically, after annotation, a first face image subset and a second face image subset are obtained. The first face images contained in the first face image subset are annotated with face detection boxes, while the second face images contained in the second face image subset are not annotated with face detection boxes.

[0038] S2.2, the first subset of face images with marked face detection boxes is divided into training set, validation set and test set in an 8:1:1 ratio.

[0039] S2.3, the YOLOV11-seg model is the base model, and the model weights are pre-trained on the publicly available Wideface dataset.

[0040] S2.4, using the distilled SAM segmentation results, a YoloV11-seg model is trained using a subset of the first face image. This embodiment of the invention provides a specific implementation method for training the YoloV11-seg model, including: (1) Perform strong enhancement processing and weak enhancement processing on the first face image to obtain a strongly enhanced face image and a weakly enhanced face image. The weak enhancement processing can be image processing such as rotation, cropping, and flipping, while the strong enhancement processing can be understood as: performing image processing such as color perturbation and stretching on the basis of the weak enhancement processing.

[0041] In this embodiment of the invention, for each first face image Regarding the first facial image Strong enhancement processing is performed to obtain a strongly enhanced face image. Regarding the first facial image Weak enhancement processing is performed to obtain a weakly enhanced face image. The two enhancements are independent of each other.

[0042] (2) Based on the weakly enhanced face image, determine the segmentation result of the strongly enhanced face instance corresponding to the strongly enhanced face image, including: (2.1) Weak enhancement processing is performed on the face detection boxes annotated in the first face image, and the face detection boxes after weak enhancement processing are horizontalized to obtain weakly enhanced face detection boxes.

[0043] In this embodiment of the invention, the face detection bounding box annotated in the first face image undergoes the same weak enhancement processing as described above. The weakly enhanced face detection bounding box is then rotated inversely by the center point and the rotation angle during enhancement to achieve horizontalization, resulting in a weakly enhanced face detection bounding box. .

[0044] (2.2) Based on the SAM architecture-based image segmentation model, weakly enhanced face instance segmentation results are generated based on weakly enhanced face images and weakly enhanced face detection boxes.

[0045] In this embodiment of the invention, the weakly enhanced face image is... and its corresponding weakly enhanced face detection bounding box The data is fed into the Segment-Anything-Model to obtain face segmentation in the image under weak enhancement, denoted as the weakly enhanced face instance segmentation result. .

[0046] (2.3) Determine the affine transformation matrix between the strongly enhanced face image and the weakly enhanced face image.

[0047] (2.4) Based on the affine transformation matrix and the weakly enhanced face instance segmentation result, determine the strongly enhanced face instance segmentation result corresponding to the strongly enhanced face image.

[0048] In this embodiment of the invention, based on the affine transformation matrix, the weakly enhanced face instance segmentation result... Inferring the results of strongly enhanced face instance segmentation .

[0049] (3) The face instance segmenter is trained using the strongly enhanced face image and its corresponding strongly enhanced face instance segmentation results.

[0050] In this embodiment of the invention, the segmentation results of the strongly enhanced face image and its corresponding strongly enhanced face instance are used. The YoloV11-seg model is trained to obtain the final face instance segmenter.

[0051] S2.5 Select the face instance segmenter with the highest mAP on the validation set, test the accuracy of the face instance segmenter on the test set, and infer detection boxes on an unlabeled second subset of face images to jointly evaluate the performance of the face instance segmenter.

[0052] S2.6 When the face instance segmenter performs well, use the current face instance segmenter; if the performance does not meet the requirements, add annotations and repeat steps S.2.4 to S.2.5.

[0053] S3 uses the YOLOv11-seg face instance segmentation model to obtain the face segmentation dataset for all collected data. Specifically, it includes: The current face instance segmenter infers face detection boxes on all unlabeled second face image subsets. The inference results and manually labeled detection boxes are saved as face detection boxes for the entire face image set. The segmented faces are then stripped of their backgrounds and saved as separate images to obtain the face segmentation dataset.

[0054] S4, Annotate facial features of a portion of the face segmentation dataset. In this embodiment of the invention, according to the project cycle and available manpower, a portion of the face segmentation data is annotated, ultimately obtaining a hybrid dataset containing partially annotated data (i.e., supervised sample dataset) and other unannotated data (i.e., unsupervised sample dataset). This hybrid dataset is then divided into training, validation, and test sets. Specifically, it includes: S4.1, the label is the type of facial feature to be detected, including normal feature class and abnormal feature class, and the total number of categories N is the number of abnormal feature classes plus one.

[0055] S4.2, add all face segmentation data without labeled facial feature types to the training set, which is called the unsupervised sample dataset.

[0056] S4.3 divides the face segmentation data labeled with facial feature types into three subsets: training set, validation set, and test set. Most of the data is divided into the training set, which is called the supervised sample dataset.

[0057] S5 modifies the unsupervised contrastive learning algorithm MoCoV2 to adapt it for semi-supervised learning. This includes modifying the model's network structure, the model's pre-pass process, the memory bank, implementing a labeling mechanism, and modifying the model's loss function to suit semi-supervised training. For details, see [link to documentation]. Figure 3 The diagram shown illustrates a backbone network of a semi-supervised training model. In the current iteration batch, the following operations are performed: S5.1, the supervised and unsupervised sample datasets are input into the target neural network. Taking MoCoV3 as an example, this network includes an improved Query network and an improved Key network. The results are obtained from the first facial feature vector and the first feature type prediction probability output by the Query network, and the first facial feature vector and the first feature type prediction probability output by the Key network.

[0058] Compared to the traditional MoCoV2 network, this embodiment of the invention adds an additional prediction head after the backbone units in both the Query and Key networks, based on the dimension dim of the first facial feature vector output by the projection head and the total number N of labeled facial feature types. The prediction head contains a single-layer MLP and a softmax function. Specifically, the improved Query network and the improved Key network have the same network structure, both including backbone units. Projector head and prediction head Backbone Unit The input consists of a supervised sample dataset and an unsupervised sample dataset, and the output is the initial feature vector; projection head The input is the initial feature vector, and the output is the first facial feature vector with dimension dim; predict the head. The input is the initial feature vector, and the output is the predicted probability of the first feature type. Based on the above network structure, the following operation is performed on each sample in both the supervised and unsupervised sample datasets: After performing different image enhancement processes on the sample, the samples are input into the Query network and the Key network, respectively. The backbone network in the Query network extracts the initial feature vector corresponding to the image-enhanced sample. The projection head in the Query network generates a first facial feature vector based on the initial feature vector. The prediction head in the Query network generates a first feature type prediction probability based on the initial feature vector. Similarly, the backbone network in the Key network extracts the initial feature vector corresponding to another image-enhanced sample. The projection head in the Key network generates a first facial feature vector based on the initial feature vector. The prediction head in the Key network generates a first feature type prediction probability based on the initial feature vector.

[0059] Each iteration batch iterates over both supervised and unsupervised sample datasets simultaneously. For example: For supervised samples in the current iteration batch For the supervised sample After undergoing different image enhancement processes, the images are fed into the Query network and the Key network, respectively. The projection head in the Query network... The first facial feature vector will be output. Predicting the head The output will predict the probability of the first feature type. Projection head in Key network The first facial feature vector will be output. Predicting the head The output will predict the probability of the first feature type. ; Similarly, for the unsupervised samples in the current iteration batch Regarding the unsupervised sample After undergoing different image enhancement processes, the images are fed into the Query network and the Key network, respectively. The projection head in the Query network... The first facial feature vector will be output. Predicting the head The output will predict the probability of the first feature type. Projection head in Key network The first facial feature vector will be output. Predicting the head The output will predict the probability of the first feature type. .

[0060] S5.2, based on the first facial feature vector and the first feature type prediction probability output by the Key network for the unsupervised sample dataset, assign the first pseudo-label to the unsupervised sample dataset, and based on the first facial feature vector output by the Key network for supervised sample data, determine the quality value corresponding to the first pseudo-label of the unsupervised sample dataset.

[0061] In this embodiment of the invention, a pre-built memory bank stores historical facial feature vectors and historical feature type prediction probabilities output by the Key network for supervised and unsupervised sample datasets in historical iteration batches. That is, in each iteration, after completing gradient backpropagation to update the network parameters, the memory bank stores the gradient-free outputs of supervised and unsupervised sample data from the previous iteration, obtained through the Key network projection head G and prediction head P under image enhancement. , , , To the memory bank; for supervised data For unsupervised data, simply use the one-hot vector of the corresponding label; If its If the value is greater than the threshold T, the first pseudo-label of the sample is determined to be... The category with the highest probability is selected. Furthermore, when storing the data passed from the previous round, the oldest feature from the memory bank is deleted to ensure that the size of the memory bank is fixed at the number of vectors in M ​​rounds of iteration, and it operates on a first-in, first-out basis.

[0062] Based on the aforementioned memory bank, this embodiment of the invention further provides a specific implementation method for determining the first pseudo-label and its quality value corresponding to the unsupervised sample dataset.

[0063] In one example, the process of determining the first pseudo-label corresponding to the unsupervised sample dataset is as follows: For each unsupervised sample in the unsupervised sample dataset, the following operations are performed: (1) The first facial feature vector output by the Key network for this unsupervised sample , denoted as the first feature vector; (2) Determine the first inner product similarity between the first feature vector and the historical facial feature vectors stored in the memory bank, so as to select multiple historical facial feature vectors that match the first feature vector from the memory bank; (3) According to the first inner product similarity, perform a weighted average on the historical feature type prediction probabilities corresponding to the multiple historical facial feature vectors that match the first feature vector, and obtain the target feature type prediction probability of the unsupervised sample belonging to each facial feature type; (4) When the target feature type prediction probability and the first feature type prediction probability output by the Key network for the unsupervised sample are both greater than the preset threshold, take the facial feature type corresponding to the largest target feature type prediction probability as the first pseudo label of the unsupervised sample.

[0064] In this embodiment of the invention, after passing the first feature vector before the current iteration, the first feature vector is compared with all historical facial feature vectors in the memory bank. Calculate the inner product similarity (denoted as the first inner product similarity), and select the top K historical facial feature vectors from high to low. Predicting probabilities using corresponding historical feature types For the top K historical facial feature vectors The predicted probability of the target feature type is obtained by performing a weighted average. If the target feature type is predicted as a probability... The predicted probability of the first feature type corresponding to the first feature vector All are greater than the preset threshold Then the prediction probability of the target feature type with the largest value will be determined. The corresponding facial feature type is used as the first pseudo-label for this unsupervised sample.

[0065] In one example, the process of determining the quality value corresponding to the first pseudo-label is as follows: First, for each supervised sample in the supervised sample dataset, the following operations are performed: (1) The first facial feature vector output by the Key network for the supervised sample is processed. (2) Determine the second inner product similarity between the second feature vector and the historical facial feature vectors stored in the memory bank, so as to select multiple historical facial feature vectors that match the second feature vector from the memory bank; (3) According to the second inner product similarity, perform a weighted average on the historical feature type prediction probabilities corresponding to the multiple historical facial feature vectors that match the second feature vector to obtain the target feature type prediction probability of the supervised sample belonging to each facial feature type. For the specific process, please refer to the calculation process of the target feature type prediction probability of the unsupervised sample mentioned above. This embodiment of the present invention will not repeat the details.

[0066] Then, based on the predicted probability of the target feature type corresponding to multiple supervised samples in the supervised sample dataset and the average of the KL divergence among the labeled facial feature types, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined. Specifically, for each supervised sample data in the current iteration, its predicted probability of the target feature type is calculated. And its true label probability The KL divergence is calculated by averaging the KL divergences of all supervised sample data. This value is then used to calculate the quality value predicted by the first pseudo-label in this iteration. The formula is shown below: ; ; ; in, Predicting the probability of target feature type in the process of pseudo-labeling for supervised sample data. Probability of True Labels KL divergence between them This represents the total number of facial feature types. Indicates the predicted probability of the target feature type. Indicates the probability of a true label. Let KL divergence mean be the average divergence. This represents the total number of supervised sample data used in this iteration. This represents the quality value of the first pseudo-label. The meaning of the above formula is as follows: The first line calculates the prediction probability of the target feature type for each supervised sample in the current iteration during the pseudo-labeling process. Probability of True Labels The smaller the KL divergence value, the more accurate the pseudo-labeling prediction; the second line calculates the average KL divergence of the currently used supervised sample data. The third line calculates the quality value predicted by the pseudo-labeling process. This value is greater than 0 and increases with the increase in prediction quality, with a maximum value of 1, indicating that the prediction is completely consistent with the label.

[0067] This completes the pseudo-labeling of all samples in the memory bank and all unsupervised samples read in the current iteration; note that samples with insufficient confidence will not be assigned the first pseudo-label.

[0068] S5.3, based on the first pseudo-label of the unsupervised sample dataset, the true label of the supervised sample dataset, and the quality value corresponding to the first pseudo-label, determine the target loss value. Use this target loss value to update the network parameters of the Query network, and then use the network parameters of the Query network to update the network parameters of the Key network. Specifically, this includes: (1) Determine the semi-supervised contrastive learning loss value based on the first pseudo-label of the unsupervised sample dataset and the true label of the supervised sample dataset. This includes the following steps: (1.1) Construct a first feature vector list based on the first facial feature vector output by the Query network. In this embodiment of the invention, the first facial feature vector output by the Query network projection head G is generated from supervised sample data and unsupervised sample data in the current iteration. , As a query_list, its label status is either a true label, the first pseudo label, or no label, depending on the pseudo-labeling process. It should be noted that since the face images input to the Query network and the Key network originate from the same source, the only difference being the enhancement methods used, the first pseudo label determined by the Key network's pseudo-labeling process can be directly assigned to the first facial feature vector output by the Query network's projection head G. The first facial feature vector output by the Query network cannot independently obtain the first pseudo label.

[0069] (1.2) Based on the first facial feature vector output by the Key network and the historical facial feature vectors stored in the memory bank, a second feature vector list is constructed. In this embodiment of the invention, the first facial feature vector output by the Key network is... , and all obtained from the memory bank Merge into key_list, and determine the tag status as true tag, first pseudo tag, or no tag based on the pseudo tag process.

[0070] (1.3) For each target sample labeled with a true label or a first pseudo label in the first feature vector list, determine whether the corresponding sample of all first facial feature vectors contained in the second feature vector list is labeled with the same true label or first pseudo label as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set.

[0071] In this embodiment of the invention, for each sample with a label (regardless of whether it is real or fake) in the query_list, each sample with the same label (regardless of whether it is real or fake) in the key_list is added to its positive match P+, and other samples in the key_list are added to its negative match P-; note that the sample and the sample in the key_list that originates from the same image must be positive matches.

[0072] (1.4) For a target sample in the first feature vector that is not labeled with a true label or a first pseudo label, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is the same as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set.

[0073] In this embodiment of the invention, for unlabeled samples in query_list, it is considered that the only positive match in P+ is a sample from the same image in key_list, and all the remaining samples in key_list are negative matches.

[0074] At this point, all samples in the query_list and key_list have been matched for positive and negative results.

[0075] (1.5) Determine the semi-supervised contrastive learning loss value based on the positive and negative matching sets. Specifically, the expression for the semi-supervised contrastive learning loss function is as follows: ; ; The first row represents the semi-supervised contrastive learning loss value. The first line represents the average loss for each sample in the query_list, and the second line calculates the loss for each sample in the query_list. Where P+(i) represents all positive matches of sample i in key_list in query_list. For temperature parameters, Let i be the first facial feature vector corresponding to sample i. For samples in P+(i) The corresponding first facial feature vector, For samples in key_list The corresponding first facial feature vector. Note that all numerator terms only contain positive matches, while the denominator terms include both positive and negative matches. This loss function encourages an increase in the similarity of the inner product of samples with positive matches, while decreasing the similarity of the inner product of samples with negative matches.

[0076] This successfully introduced additional positive matching from true and false labels into the contrastive learning loss function of MoCoV2.

[0077] (2) Determine the regression loss value based on the true labels of the supervised sample dataset and the predicted probabilities of their corresponding target feature types. In this embodiment of the invention, for the supervised data in the current iteration, its labels and the output of the prediction head P in the query network are used. Establish a softmax regression loss function .

[0078] (3) Using the quality value corresponding to the first pseudo-label of the unsupervised sample dataset, the regression loss value and the semi-supervised contrastive learning loss value are fused to obtain the target loss value.

[0079] In this embodiment of the invention, there will be supervised data. Introducing this, we obtain the final loss function: ; In this formula, For the target loss value, These are hyperparameters that control the ratio of the two loss functions; This represents the quality value of the pseudo-labeling process in the current iteration. The semi-supervised contrastive learning loss value is determined based on supervised and unsupervised sample data; This represents the softmax regression loss value determined based on supervised sample data. In the final loss function, as the number of iterations increases, the quality of the pseudo-labeling process improves. Increase the semi-supervised contrastive learning loss value based on the first pseudo-label. The weights also gradually increase, and the loss function is gradually dominated by the semi-supervised loss function.

[0080] (4) Update the network parameters of the Query network and the Key network using the target loss value. In one implementation, the query network calculates the gradient and updates it using gradient backpropagation, while the key network depends on the query network for momentum updates. Specifically, the gradients of all output vectors of the Query network are cut off, and the network parameters of the Key network are updated based on the updated network parameters of the Query network according to the following EMA formula: ; in For the parameters of the query network, For the parameters of the key network, The momentum update ratio controls the magnitude of each key network update.

[0081] Finally, when the current iteration batch meets the preset stopping conditions, the Query network is used as the recognition backbone network in the facial feature recognition network. Specifically, the recognition backbone network includes the backbone units of the Query network and the projection head.

[0082] S6, train the recognition backbone network on the training set using the modified algorithm. That is, call the training algorithm provided in S5 above to train the recognition backbone network. The parameters during the training process can be determined according to the following steps: S6.1 Find suitable batch size and memory bank size. Specifically: (1) The batch size and memory bank size determine the number of positive and negative pairs in each batch. Both are proportional to the model performance, so they should be as large as possible within the limits of available video memory. The memory bank size is an integer multiple of the batch size.

[0083] (2) Select a batch size according to the GPU model and quantity, and set the memory bank size to an integer multiple of the batch size and significantly greater than 1; this is because the query data read in the batch size needs to be input into the network for inference and gradient descent, while the key in the memory bank only needs to be used as a constant in loss calculation, so it does not occupy too much GPU memory.

[0084] (3) Use the selected batch size and memory bank size to train the model. If the memory bank is not filled in the first few iterations, only Lsoft is calculated.

[0085] (4) If an over-memory error occurs, reduce the batch size and memory bank size until the video memory no longer reports an error.

[0086] S6.2 Select cosine learning rate for update, set the maximum number of iterations equal to the number of epochs, the initial learning rate to 0.001, and the minimum learning rate to 0.00001 for training.

[0087] S6.3 Calculate the average loss every specific number of epochs (greater than 50) until training is terminated when the average loss stops decreasing.

[0088] S7, the model is frozen to identify the backbone network. Using labeled training data and high-confidence pseudo-labels from the semi-supervised learning process, a multi-class prediction head is trained to obtain multiple templates for each facial feature and the absence of any feature, stored as a facial feature template vector library. The multi-class prediction head is the facial feature template vector library, containing multiple templates for each facial feature type. Facial feature types include normal feature classes and one or more abnormal feature classes, such as... Figure 4 The diagram illustrates a feature vector template for a specific condition. Specifically, a multi-subclass prediction head is created to represent multiple templates W(dim, M) for each facial feature, where M = ( * + Where dim is the dimension of the feature vector, and M is the total number of templates. The number of abnormal feature categories to be detected. The number of feature templates for each abnormal feature class. Number of feature templates for each normal feature class: ; ; ; ; in, It is a set of templates for all abnormal feature classes. It is a set of templates for all normal feature classes. This is a set comprising all facial feature types. Note Significantly greater than This is because the proportion of normal facial symptoms in images collected in medical settings should be higher than the proportion of symptomatic symptoms. The specific ratio can be adjusted according to the probability of occurrence of each facial feature.

[0089] Based on this, the specific process of training the subclass prediction head is as follows: (1) By identifying the backbone network, the second facial feature vector corresponding to the unsupervised sample data is determined, and the prediction head in the Query network is used to predict the second feature type prediction probability of the unsupervised sample dataset belonging to each facial feature type. The second pseudo label is assigned to the unsupervised sample dataset based on the second feature type prediction probability. For example, if the largest second feature type prediction probability is greater than the threshold, the second pseudo label is assigned to the unsupervised sample data.

[0090] (2) Mix the unsupervised data with the second pseudo-label and the supervised data to form a dataset for training the subclass prediction head. Each sample in the dataset has a unique label, and the probability of occurrence of each facial feature type is calculated. .

[0091] (3) During training, the images of the current batch are fed into the backbone recognition to obtain the second facial feature vector and calculate its cosine similarity with each template. : ; ; ; Where B is the batch size. That is, unsupervised sample data With template Cosine similarity between them.

[0092] (4) Determine the cosine similarity between the second facial feature vector and the template corresponding to each abnormal feature class, and take the largest cosine similarity as the first cosine similarity between the second facial feature vector and each abnormal feature class. Specifically, for N abnormal feature classes, take the corresponding... The maximum cosine similarity of the templates is the cosine similarity S1(B,N) of the feature class: ; ; .

[0093] (5) Determine the cosine similarity between the second facial feature vector and the template corresponding to the normal feature class, and take the largest cosine similarity as the second cosine similarity between the second facial feature vector and the normal feature class. Specifically, for the normal feature class, take the maximum cosine similarity of the corresponding multiple templates as the cosine similarity S2(B,1): ; .

[0094] (6) Concatenate the first cosine similarity and the second cosine similarity to obtain the target cosine similarity. Specifically, concatenate S1 and S2 to obtain the cosine similarity S between each sample and N facial feature classes and facial featureless classes. .

[0095] (7) Based on the probability of occurrence of each facial feature type The loss value of the prediction head is determined by combining the target cosine similarity, and this loss value is used to update the network parameters of the multi-subclass prediction head. Specifically, the loss is calculated using a loss function, and the model is iteratively optimized. The loss function for each sample is: ; ; The loss function `log` internally uses the loss function of the ArcFace algorithm, where... The angle between the target sample and its corresponding category; Sample The labeling category; The value is a constant, and the mapping angle to the radius is... A sphere; As a constant, compressing the vector angle between the target sample and its labeled category improves intra-class compactness; The probability of occurrence of each facial feature type in (2) is a reweighted parameter designed based on the imbalance of data in the scene, which is used to enhance the loss of rare categories and reduce the loss of common categories. for The number of times a category appears in the entire dataset.

[0096] The loss no longer decreases when the training model reaches the validation set. The resulting multi-subclass predicted head feature vectors are multi-subclass feature templates for each facial feature class and the facial no-feature class.

[0097] Furthermore, the embodiments of the present invention provide an explanation of the data reasoning portion, including: S8 uses the trained face instance segmenter to obtain face segmentation. That is, it acquires the face image to be processed; and uses the face instance segmenter to extract the face segmentation result corresponding to the face image to be processed.

[0098] S9 inputs the face segmentation data into the recognition backbone network of the facial feature recognition network to extract the facial feature vector. That is, the third facial feature vector X(B, dim) corresponding to the face segmentation result is extracted through the recognition backbone network of the facial feature recognition network.

[0099] S10 compares the facial feature vector with the facial feature template vector library. When the maximum similarity is greater than a threshold, a match is considered successful, and the predicted facial features are output. That is, through the multi-subclass prediction head in the facial feature recognition network, the third facial feature vector corresponding to the face segmentation result is compared with the facial feature template vector library to obtain the facial features corresponding to the face image to be processed. Specifically, this includes: calculating the cosine similarity between the obtained facial feature vector and all feature templates for each facial feature; and calculating the cosine similarity S between the obtained feature vector to be matched and each facial feature, including those without features. The softmax function is used to obtain the feature vector to be matched and the normalized confidence score of each facial feature. : ; ; In this formula, B represents the number of images used in this inference, N represents the number of abnormal feature categories, and +1 represents the normal feature category; Let P be the cosine similarity between each image and each class to be matched; P is the probability that the vector to be matched belongs to all classes, and its value is 1.

[0100] When the maximum normalized confidence exceeds the threshold, the feature vector is matched as the facial feature class.

[0101] In summary, compared to models obtained through conventional supervised training, this model achieves accurate face segmentation by using labeled detection boxes and a large-scale segmentation model (SAM) for distillation, thus reducing the impact of environmental noise in the image. More importantly, this model fully utilizes semi-supervised training and supervised multi-class head fine-tuning to extract information from unlabeled data, improving model performance with the same amount of labeled data.

[0102] Based on the foregoing embodiments, this invention provides a facial feature recognition device based on semi-supervised pre-training, see [link to previous embodiment]. Figure 5 The diagram shows a structural schematic of a facial feature recognition device based on semi-supervised pre-training. The device mainly includes the following parts: The face segmentation module 502 is used to generate a face segmentation dataset based on a set of face images in the target scene; The dataset partitioning module 504 is used to divide the face segmentation dataset into a supervised sample dataset and an unsupervised sample dataset. The supervised samples contained in the supervised sample dataset are labeled with true labels, and the true labels are the facial feature types to which the supervised samples belong. The feature recognition network training module 506 is used to perform semi-supervised pre-training on the recognition backbone network of the facial feature recognition network using supervised sample datasets and unsupervised sample datasets. After pre-training, the parameters of the recognition backbone network are frozen, and the multi-subclass prediction head in the facial feature recognition network is trained using supervised sample datasets, unsupervised sample datasets, and pseudo-labels generated for the unsupervised sample datasets to obtain the target facial feature recognition network, which is used to predict the facial features of the target object.

[0103] The facial feature recognition device based on semi-supervised pre-training provided in this invention uses semi-supervised training to train the recognition backbone network, and then uses supervised training to train the multi-subclass prediction head after freezing the recognition backbone network. This fully utilizes the information in the unsupervised sample dataset, improves the model training performance under the same number of annotations, and overcomes the problems of high cost of annotation data and low accuracy of trained models in medical scenarios for facial feature recognition.

[0104] In one implementation, the face segmentation module 502 is specifically used for: Strong enhancement and weak enhancement processing are performed on the first face image with face detection boxes in the face image set to obtain strong enhancement face image and weak enhancement face image respectively; Based on weakly enhanced face images, determine the segmentation results of strongly enhanced face instances corresponding to strongly enhanced face images; The face instance segmenter is trained using strongly enhanced face images and their corresponding strongly enhanced face instance segmentation results; Using the trained face instance segmenter, face segmentation is performed on the first face image with face detection boxes labeled and the second face image without face detection boxes labeled in the face image set, resulting in a generated face segmentation dataset.

[0105] In one implementation, the face segmentation module 502 is specifically used for: Weak enhancement processing is applied to the face detection bounding boxes annotated in the first face image, and the weakly enhanced face detection bounding boxes are then horizontalized to obtain weakly enhanced face detection bounding boxes. Using an image segmentation model based on the SAM architecture, weakly enhanced face instance segmentation results are generated based on weakly enhanced face images and weakly enhanced face detection boxes. Determine the affine transformation matrix between the strongly enhanced face image and the weakly enhanced face image; Based on the affine transformation matrix and the weakly enhanced face instance segmentation results, the strongly enhanced face instance segmentation results corresponding to the strongly enhanced face images are determined.

[0106] In one implementation, the feature recognition network training module 506 is specifically used for: Perform the following operations in the current iteration batch: The supervised sample dataset and the unsupervised sample dataset are input into the target neural network, which includes an improved Query network and an improved Key network, to obtain the first facial feature vector and the first feature type prediction probability output by the Query network, and the first facial feature vector and the first feature type prediction probability output by the Key network. Based on the first facial feature vector and the first feature type prediction probability output by the Key network for the unsupervised sample dataset, the first pseudo-label is assigned to the unsupervised sample dataset. Based on the first facial feature vector output by the Key network for supervised sample data, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined. Based on the first pseudo-label labeled in the unsupervised sample dataset, the true label labeled in the supervised sample dataset, and the quality value corresponding to the first pseudo-label, a target loss value is determined. The network parameters of the Query network are updated using the target loss value, and the network parameters of the Key network are updated using the network parameters of the Query network. The true label is the facial feature type to which the supervised sample belongs. When the current iteration batch meets the preset stopping conditions, the Query network will be used as the recognition backbone network in the facial feature recognition network.

[0107] In one implementation, both the Query network and the Key network include a backbone unit, a projection head, and a prediction head; the recognition backbone network includes the backbone unit and projection head in the Query network; the feature recognition network training module 506 is specifically used for: For each sample in the supervised and unsupervised sample datasets of the current iteration, perform the following operation: After performing different image enhancement processes on the sample, the images were input into the Query network and the Key network, respectively. The initial feature vector corresponding to the sample after image enhancement is extracted through the backbone network in the Query network. The first facial feature vector is generated based on the initial feature vector through the projection head in the Query network. The first feature type prediction probability is generated based on the initial feature vector through the prediction head in the Query network. Furthermore, the initial feature vector corresponding to another image-enhanced sample is extracted through the backbone network in the Key network, the first facial feature vector is generated based on the initial feature vector through the projection head in the Key network, and the first feature type prediction probability is generated based on the initial feature vector through the prediction head in the Key network.

[0108] In one implementation, a pre-built memory bank stores historical facial feature vectors and historical feature type prediction probabilities output by the Key network for supervised and unsupervised sample datasets under historical iteration batches; the feature recognition network training module 506 is specifically used for: For each unsupervised sample in the unsupervised sample dataset of the current iteration, perform the following operation: The first facial feature vector output by the Key network for this unsupervised sample is denoted as the first feature vector. Determine the first inner product similarity between the first feature vector and the historical facial feature vectors stored in the memory bank, so as to filter out multiple historical facial feature vectors that match the first feature vector from the memory bank; Based on the first inner product similarity, the predicted probability of the historical feature type corresponding to multiple historical facial feature vectors matched by the first feature vector is weighted and averaged to obtain the predicted probability of the target feature type of the unsupervised sample belonging to each facial feature type. If both the target feature type prediction probability and the first feature type prediction probability output by the Key network for the unsupervised sample are greater than a preset threshold, the facial feature type corresponding to the highest target feature type prediction probability will be used as the first pseudo-label of the unsupervised sample.

[0109] In one implementation, the feature recognition network training module 506 is specifically used for: For each supervised sample in the supervised sample dataset of the current iteration, perform the following operation: The first facial feature vector output by the Key network for this supervised sample is denoted as the second feature vector; Determine the second inner product similarity between the second feature vector and the historical facial feature vectors stored in the memory bank, in order to filter out multiple historical facial feature vectors that match the second feature vector from the memory bank; Based on the second inner product similarity, the predicted probability of the historical feature type corresponding to multiple historical facial feature vectors matched by the second feature vector is weighted and averaged to obtain the predicted probability of the target feature type of the supervised sample belonging to each facial feature type. Based on the predicted probability of the target feature type corresponding to each supervised sample in the supervised sample dataset in this round of iteration, and the average value of the KL divergence between the labeled facial feature types, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset in this round of iteration is determined.

[0110] In one implementation, the feature recognition network training module 506 is specifically used for: The semi-supervised contrastive learning loss value is determined based on the first pseudo-label labeled in the unsupervised sample dataset and the true label labeled in the supervised sample dataset; and the regression loss value is determined based on the true label labeled in the supervised sample dataset and the predicted probability of the corresponding target feature type. The target loss value is obtained by fusing the regression loss value and the semi-supervised contrastive learning loss value with the quality value corresponding to the first pseudo-label of the unsupervised sample dataset.

[0111] In one implementation, the feature recognition network training module 506 is specifically used for: Construct a list of first feature vectors based on the first facial feature vector output by the Query network; A second feature vector list is constructed based on the first facial feature vector output by the Key network and the historical facial feature vectors stored in the memory bank. For a target sample labeled with a true label or a first pseudo label in the first feature vector list, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is labeled with the same true label or first pseudo label as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. For a target sample in the first feature vector that is not labeled with a true label or a first pseudo label, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is the same as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. The semi-supervised contrastive learning loss value is determined based on the positive and negative matching sets.

[0112] In one implementation, the multi-subclass prediction head is a facial feature template vector library, which contains multiple templates for each facial feature type. The facial feature types include normal feature classes and one or more abnormal feature classes. The feature recognition network training module 506 is specifically used for: By identifying the backbone network, the second facial feature vector corresponding to the unsupervised sample data is determined, and the prediction head in the Query network is used to predict the second feature type prediction probability of the unsupervised sample dataset belonging to each facial feature type. Based on the second feature type prediction probability, the unsupervised sample dataset is assigned a second pseudo label. Determine the cosine similarity between the second facial feature vector and the template corresponding to each abnormal feature class, and take the largest cosine similarity as the first cosine similarity between the second facial feature vector and each abnormal feature class; Determine the cosine similarity between the second facial feature vector and the template corresponding to the normal feature class, and take the largest cosine similarity as the second cosine similarity between the second facial feature vector and the normal feature class; The first cosine similarity and the second cosine similarity are concatenated to obtain the target cosine similarity. Based on the second pseudo-label corresponding to the unsupervised sample dataset and the true label labeled in the supervised sample dataset, the occurrence probability of each facial feature type is calculated, and the prediction head loss value is determined by combining the target cosine similarity. The prediction head loss value is then used to update the network parameters of the multi-subclass prediction head.

[0113] In one embodiment, a facial feature recognition module is further included, for: Obtain the face image to be processed; The face segmentation result corresponding to the face image to be processed is extracted using a face instance segmenter. The third facial feature vector corresponding to the face segmentation result is extracted through the recognition backbone network in the facial feature recognition network. Then, the third facial feature vector corresponding to the face segmentation result is compared with the facial feature template vector library through the multi-subclass prediction head in the facial feature recognition network to obtain the facial features corresponding to the face image to be processed.

[0114] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0115] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A facial feature recognition method based on semi-supervised pre-training, characterized in that, include: Generate a face segmentation dataset based on a set of face images from the target scene; The face segmentation dataset is divided into a supervised sample dataset and an unsupervised sample dataset. Using the supervised sample dataset and the unsupervised sample dataset, the recognition backbone network in the facial feature recognition network is semi-supervised pre-trained. After the pre-training is completed, the parameters of the recognition backbone network are frozen. Then, using the supervised sample dataset, the unsupervised sample dataset, and the pseudo-labels generated for the unsupervised sample dataset, the multi-subclass prediction head in the facial feature recognition network is trained to obtain the target facial feature recognition network, which is used to predict the facial features of the target object.

2. The facial feature recognition method based on semi-supervised pre-training according to claim 1, characterized in that, A face segmentation dataset is generated based on a set of face images from the target scene, including: The first face image with a face detection box marked in the face image set is subjected to strong enhancement processing and weak enhancement processing respectively to obtain a strongly enhanced face image and a weakly enhanced face image. Based on the weakly enhanced face image, determine the segmentation result of the strongly enhanced face instance corresponding to the strongly enhanced face image; The face instance segmenter is trained using the strongly enhanced face image and its corresponding strongly enhanced face instance segmentation result; Using the trained face instance segmenter, face segmentation is performed on the first face image with face detection boxes labeled and the second face image without face detection boxes labeled in the face image set, to obtain a generated face segmentation dataset.

3. The facial feature recognition method based on semi-supervised pre-training according to claim 2, characterized in that, Based on the weakly enhanced face image, determine the segmentation result of the strongly enhanced face instance corresponding to the strongly enhanced face image, including: The weak enhancement processing is applied to the face detection box annotated in the first face image, and the weakly enhanced face detection box is then horizontalized to obtain a weakly enhanced face detection box. Using an image segmentation model based on the SAM architecture, weakly enhanced face instance segmentation results are generated based on the weakly enhanced face image and the weakly enhanced face detection box. Determine the affine transformation matrix between the strongly enhanced face image and the weakly enhanced face image; Based on the affine transformation matrix and the weakly enhanced face instance segmentation result, the strongly enhanced face instance segmentation result corresponding to the strongly enhanced face image is determined.

4. The facial feature recognition method based on semi-supervised pre-training according to claim 1, characterized in that, Using the supervised sample dataset and the unsupervised sample dataset, semi-supervised pre-training is performed on the recognition backbone network in the facial feature recognition network, including: Perform the following operations in the current iteration batch: The supervised sample dataset and the unsupervised sample dataset are input into the target neural network, which includes an improved Query network and an improved Key network to obtain the first facial feature vector and the first feature type prediction probability output by the Query network, and the first facial feature vector and the first feature type prediction probability output by the Key network. Based on the first facial feature vector and the first feature type prediction probability output by the Key network for the unsupervised sample dataset, a first pseudo-label is assigned to the unsupervised sample dataset, and based on the first facial feature vector output by the Key network for the supervised sample data, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined. Based on the first pseudo-label labeled in the unsupervised sample dataset, the true label labeled in the supervised sample dataset, and the quality value corresponding to the first pseudo-label, a target loss value is determined. The network parameters of the Query network are updated using the target loss value, and the network parameters of the Key network are updated using the network parameters of the Query network. The true label is the facial feature type to which the supervised sample belongs. When the current iteration batch meets the preset stopping condition, the Query network is used as the recognition backbone network in the facial feature recognition network.

5. The facial feature recognition method based on semi-supervised pre-training according to claim 4, characterized in that, Both the Query network and the Key network include a backbone unit, a projection head, and a prediction head. The recognition backbone network includes the backbone unit and the projection head in the Query network. The supervised sample dataset and the unsupervised sample dataset are input into the target neural network, which includes an improved Query network and an improved Key network, to obtain the first facial feature vector and the first feature type prediction probability output by the Query network, and the first facial feature vector and the first feature type prediction probability output by the Key network, including: For each sample in the supervised sample dataset and the unsupervised sample dataset in the current iteration, perform the following operation: After performing different image enhancement processes on the sample, the data is input into the Query network and the Key network, respectively. The initial feature vector corresponding to the sample after image enhancement is extracted through the backbone network in the Query network. A first facial feature vector is generated based on the initial feature vector through the projection head in the Query network. A first feature type prediction probability is generated based on the initial feature vector through the prediction head in the Query network. Furthermore, through the backbone network in the Key network, an initial feature vector corresponding to another image-enhanced sample is extracted; through the projection head in the Key network, a first facial feature vector is generated based on the initial feature vector; and through the prediction head in the Key network, a first feature type prediction probability is generated based on the initial feature vector.

6. The facial feature recognition method based on semi-supervised pre-training according to claim 4, characterized in that, The pre-built memory stores historical facial feature vectors and historical feature type prediction probabilities output by the Key Network for the supervised sample dataset and the unsupervised sample dataset in historical iteration batches; based on the first facial feature vector and the first feature type prediction probability output by the Key Network for the unsupervised sample dataset, a first pseudo-label is assigned to the unsupervised sample dataset, including: For each unsupervised sample in the unsupervised sample dataset of the current iteration, perform the following operation: The first facial feature vector output by the Key network for this unsupervised sample is denoted as the first feature vector; Determine the first inner product similarity between the first feature vector and the historical facial feature vectors stored in the memory bank, so as to filter out a plurality of historical facial feature vectors that match the first feature vector from the memory bank; Based on the first inner product similarity, the predicted probability of the historical feature type corresponding to the multiple historical facial feature vectors matched by the first feature vector is weighted and averaged to obtain the predicted probability of the target feature type of the unsupervised sample belonging to each of the facial feature types. If both the predicted probability of the target feature type and the predicted probability of the first feature type output by the Key network for the unsupervised sample are greater than a preset threshold, the facial feature type corresponding to the highest predicted probability of the target feature type is taken as the first pseudo-label of the unsupervised sample.

7. The facial feature recognition method based on semi-supervised pre-training according to claim 6, characterized in that, Based on the first facial feature vector output by the Key network for the supervised sample data, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset is determined, including: For each supervised sample in the current iteration, perform the following operation: The first facial feature vector output by the Key network for this supervised sample is denoted as the second feature vector; Determine the second inner product similarity between the second feature vector and the historical facial feature vectors stored in the memory bank, so as to filter out multiple historical facial feature vectors that match the second feature vector from the memory bank; Based on the second inner product similarity, the weighted average of the historical feature type prediction probabilities corresponding to the multiple historical facial feature vectors matched by the second feature vector is used to obtain the target feature type prediction probability of the supervised sample belonging to each of the facial feature types. Based on the predicted probability of the target feature type corresponding to each supervised sample in the supervised sample dataset in the current iteration, and the average value of the KL divergence between the labeled facial feature types, the quality value corresponding to the first pseudo-label of the unsupervised sample dataset in the current iteration is determined.

8. The facial feature recognition method based on semi-supervised pre-training according to claim 6, characterized in that, The target loss value is determined based on the first pseudo-label labeled in the unsupervised sample dataset, the true label labeled in the supervised sample dataset, and the quality value corresponding to the first pseudo-label, including: Based on the first pseudo-label labeled in the unsupervised sample dataset and the true label labeled in the supervised sample dataset, a semi-supervised contrastive learning loss value is determined; and based on the true label labeled in the supervised sample dataset and its corresponding target feature type prediction probability, a regression loss value is determined. The regression loss value and the semi-supervised contrastive learning loss value are fused using the quality value corresponding to the first pseudo-label in the unsupervised sample dataset to obtain the target loss value.

9. The facial feature recognition method based on semi-supervised pre-training according to claim 8, characterized in that, Based on the first pseudo-label labeled in the unsupervised sample dataset and the true label labeled in the supervised sample dataset, a semi-supervised contrastive learning loss value is determined, including: Based on the first facial feature vector output by the Query network, a first feature vector list is constructed; Based on the first facial feature vector output by the Key network and the historical facial feature vectors stored in the memory bank, a second feature vector list is constructed. For a target sample labeled with the true label or the first pseudo label in the first feature vector list, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is labeled with the same true label or the first pseudo label as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. For a target sample in the first feature vector that is not labeled with the true label or the first pseudo label, determine whether the sample corresponding to the first facial feature vector contained in the second feature vector list is the same as the target sample. If yes, add the first facial feature vector to the positive matching set; otherwise, add the first facial feature vector to the negative matching set. The semi-supervised contrastive learning loss value is determined based on the positive matching set and the negative matching set.

10. The facial feature recognition method based on semi-supervised pre-training according to claim 1, characterized in that, The multi-subclass prediction head is a facial feature template vector library, which contains multiple templates for each facial feature type, including normal feature classes and one or more abnormal feature classes. The multi-subclass prediction head in the facial feature recognition network is trained using the supervised sample dataset, the unsupervised sample dataset, and pseudo-labels generated for the unsupervised sample dataset, including: The second facial feature vector corresponding to the unsupervised sample data is determined through the recognition backbone network, and the second feature type prediction probability of the unsupervised sample dataset belonging to each of the facial feature types is predicted through the prediction head in the Query network, so as to assign a second pseudo label to the unsupervised sample dataset based on the second feature type prediction probability. Determine the cosine similarity between the second facial feature vector and the template corresponding to each of the abnormal feature classes, and take the largest cosine similarity as the first cosine similarity between the second facial feature vector and each of the abnormal feature classes; Determine the cosine similarity between the second facial feature vector and the template corresponding to the normal feature class, and take the largest cosine similarity as the second cosine similarity between the second facial feature vector and the normal feature class; The first cosine similarity and the second cosine similarity are concatenated to obtain the target cosine similarity; Based on the second pseudo-label corresponding to the unsupervised sample dataset and the true label labeled in the supervised sample dataset, the occurrence probability of each facial feature type is calculated, and the prediction head loss value is determined in combination with the target cosine similarity, so as to update the network parameters of the multi-subclass prediction head using the prediction head loss value.

11. A facial feature recognition device based on semi-supervised pre-training, characterized in that, include: The face segmentation module is used to generate a face segmentation dataset based on a set of face images in a target scene. The dataset partitioning module is used to divide the face segmentation dataset into a supervised sample dataset and an unsupervised sample dataset. The feature recognition network training module is used to perform semi-supervised pre-training on the recognition backbone network of the facial feature recognition network using the supervised sample dataset and the unsupervised sample dataset. After the pre-training is completed, the parameters of the recognition backbone network are frozen, and the multi-subclass prediction head of the facial feature recognition network is trained using the supervised sample dataset, the unsupervised sample dataset, and pseudo-labels generated for the unsupervised sample dataset to obtain the target facial feature recognition network, which is used to predict the facial features of the target object.

Images