Ear brushing identity recognition method based on pseudo tag semi-supervised learning

Abstract

The invention belongs to the field of identity recognition, and discloses an ear brushing identity recognition method based on pseudo-label semi-supervised learning, which comprises the following steps: 1, constructing a data set for training and labeling the data set to obtain a labeled training set, a non-labeled training set and a labeled verification set; s2, determining a to-be-used convolutional neural network model, and modifying a training process and a detection process of the to-be-used convolutional neural network model according to an improved algorithm and an improved recognition principle; s3, training the convolutional neural network model by adopting an improved Mean Teaching algorithm to obtain a trained neural network model; s4, using the trained neural network model to calculate a to-be-recognized ear image to obtain a feature vector; and S5, adopting a nearest neighbor search algorithm to match the feature vector obtained in the step S4 with a human ear feature vector which is input into a database and needs to be used for identification, and realizing end-to-end human ear identity identification. According to the method, the detection performance of small targets and tail categories is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of identity recognition, and in particular to an ear brushing identity recognition method based on pseudo-label semi-supervised learning. Background Art

[0002] With the continuous advancement of deep learning technology, facial and ear biometrics have become a key component of modern security systems. Facial recognition technology has been widely used in many fields and has become increasingly mature, thanks to its target prominence and rich data sets. However, ear recognition, as an emerging field of biometrics, has shown great application potential and theoretical value due to its unique physiological structure and easy collection characteristics.

[0003] Traditional ear recognition methods, such as the twelve-point ear measurement method, mainly rely on manual recognition, which is not only inefficient but also susceptible to subjectivity. With the introduction of neural network algorithms in the second half of the 20th century, ear recognition technology began to develop in the direction of automation and intelligence. In the 21st century, researchers have significantly improved the accuracy and robustness of recognition through principal component analysis (PCA), kernel principal component analysis (KPCA), and support vector machines. Although traditional machine learning methods have made significant progress in the field of ear recognition, with the vigorous development of deep learning technology, researchers have begun to explore its application potential in ear recognition. This transformative technology has greatly promoted the improvement of recognition accuracy and efficiency. The introduction of deep learning, especially convolutional neural networks (CNN), provides powerful feature extraction capabilities for automatic recognition and reduces the reliance on manual feature engineering. Similarly, deep learning has also been applied to ear recognition. By designing a network architecture suitable for ear characteristics, such as increasing network depth and adopting multi-scale feature fusion, deep learning models have surpassed traditional methods on public datasets, showing their superiority in ear recognition tasks.

[0004] The research on ear recognition technology started late and the research scale is small. At present, many ear detection methods are inspired by facial detection methods. The main challenges faced by ear recognition include:

[0005] (1) For the model: The ear is very small in the original image. Whether measured in terms of volume or surface area, a single ear is far inferior to the face. Furthermore, mainstream object detection network architectures often include one or more image downsampling steps, which will have an adverse effect on the performance and robustness of the ear detector.

[0006] (2) Regarding data: Relevant training data resources are relatively scarce, and a standardized ear database like the facial database has not yet been established. When training ear detection models, deep learning relies heavily on a large number of training samples. Therefore, when the number of available samples is limited, the model is prone to overfitting. Overfitting means that when evaluating model performance, the model performs well on the training data, but the prediction results on the validation data set or test data are significantly different from expectations, indicating that the model learning overfits the training data, resulting in a decrease in its generalization ability and difficulty in using it for content outside the training data. Summary of the invention

[0007] The purpose of the present invention is to disclose an ear brushing identity recognition method based on pseudo-label semi-supervised learning to solve the technical problems raised in the background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention provides an ear brushing identity recognition method based on pseudo-label semi-supervised learning, comprising:

[0010] S1, construct a data set for training and annotate the data set to obtain a labeled training set, an unlabeled training set, and a labeled validation set;

[0011] S2, determine the convolutional neural network model to be used, and modify its training process and detection process according to the improved algorithm and recognition principle;

[0012] S3, using the improved MeanTeacher algorithm to train the convolutional neural network model to obtain a trained neural network model;

[0013] S4, using the trained neural network model to calculate the human ear image to be identified to obtain a feature vector;

[0014] S5, using the nearest neighbor search algorithm to match the feature vector obtained in S4 with the human ear feature vector that has been entered into the database and needs to be used for identification, to achieve end-to-end human ear identity recognition.

[0015] Preferably, S1 includes:

[0016] The public datasets on the Internet were searched, and then the ear images of three different sizes, small, medium and large, were fused according to the preset ratio (9:4:1) to construct the dataset.

[0017] Through manual review, we remove erroneous images, duplicate images, and images that are too difficult to identify in the constructed dataset; erroneous images include images that contain ears but are not human ears;

[0018] The processed data set is divided into three parts: a labeled training set, an unlabeled training set, and a labeled validation set; the number of images in the labeled training set is 1% of the total number of ear images in the data set;

[0019] LabelImg software is used to annotate the data of the labeled training set and the labeled validation set. The human ear area in the image is framed with a target box, and the framed area is labeled with the ear label.

[0020] Preferably, the image with too high recognition difficulty is an image without ear contour.

[0021] Preferably, the convolutional neural network model includes a Faster-RCNN model.

[0022] Preferably, S3 includes:

[0023] In the first iteration, the following operations are performed:

[0024] Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b;

[0025] Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image;

[0026] Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence to generate a pseudo label data set; store the pseudo label data set in the cache library; low confidence refers to a confidence level less than 0.85;

[0027] Starting from the second iteration, the following operations are performed:

[0028] Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b;

[0029] Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image;

[0030] Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence and generate a pseudo label data set; use A to represent the pseudo label data set obtained in the current iteration, and store A in the cache library; low confidence refers to a confidence level less than 0.85;

[0031] Extract multiple images from the cache library to form a set B, where the set B does not contain the images in the set A, and the number of images in the set B is the same as the number of images in the set A;

[0032] Mix the images in set A and set B by Mixup to obtain a first image set;

[0033] Processing set A and images extracted from set B by Mosaic to obtain a second image set; the number of images extracted from set B is the same as the number of images in set A;

[0034] Performing strong enhancement processing on the images in the first image set and the second image set respectively to obtain a third image set;

[0035] Performing labeled supervised enhancement processing on the labeled image to obtain an enhanced image;

[0036] Mixing the enhanced image and the third image set as input data of the student model;

[0037] Through the supervision loss L supervised and the unsupervised loss L unsupervised Optimize the student model;

[0038] The exponential moving average strategy is used to update the teacher model.

[0039] Preferably, the weak enhancement process includes:

[0040] The first step is to load the image to be weakly enhanced;

[0041] In the second step, the image in the first step is randomly scaled;

[0042] The third step is to randomly flip the image obtained in the second step.

[0043] Preferably, the process of label-supervised enhancement processing includes:

[0044] The first step is to load the image and the corresponding annotation information;

[0045] In the second step, the image in the first step is randomly scaled;

[0046] The third step is to randomly flip the image obtained in the second step;

[0047] The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step;

[0048] The fifth step is to filter out target frames that are too small and the images where the target frames are located. Target frames that are too small are target frames in images with a size less than 80×80.

[0049] Preferably, the process of strong enhancement processing includes:

[0050] The first step is to load the image to be weakly enhanced;

[0051] In the second step, the image in the first step is randomly scaled;

[0052] The third step is to randomly flip the image obtained in the second step;

[0053] The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step;

[0054] Step 5: Apply a geometric-based random enhancement strategy to the image obtained in step 4.

[0055] Step 6: Apply a random enhancement strategy based on color_space to the image obtained in step 5;

[0056] Step 7: Apply a geometric-based random enhancement strategy to the image obtained in step 6;

[0057] In the eighth step, the target frames that are too small and the images where the target frames are located are filtered out. The target frames that are too small are the target frames in the images whose size is less than 80×80.

[0058] Preferably, for supervised learning, a smooth L1 loss function is used:

[0059]

[0060] L1(x,y) represents the smooth loss function, x and y represent the predicted value and true value of the neural network model respectively;

[0061] For unsupervised learning, use the consistency regularization loss:

[0062]

[0063] N represents the number of samples in each batch; L consistency represents the consistency regularization loss function; f(x i ) represents the output result of the student model for the i-th image; g(x i ) represents the output result of the teacher model calculating the i-th image;

[0064] In each training iteration, the parameters of the student model are updated by minimizing the total loss function, which is the weighted sum of these two losses, L total is the total loss function, L supervised To monitor the loss, L unsupervised is the unsupervised loss, weight w u Used to balance the contributions between the two:

[0065] L total =L supervised +w u *L unsupervised .

[0066] Preferably, S2 includes:

[0067] The convolutional neural network model to be used is the Faster R-CNN model;

[0068] Modify the FPN and global average pooling layers in the neck part of the Faster-RCNN model to enable the Faster-RCNN model to output the feature vector of the ear in the human ear image to be identified, and then train the modified model;

[0069] Inputting the human ear image to be identified into the trained model, and obtaining the feature vector of the ear in the human ear image output by the trained model;

[0070] The global average pooling operation is calculated as follows:

[0071]

[0072] Among them, G c represents the output value of the cth channel after global average pooling; F i,j,c Represents the pixel value of the original feature map at position (i, j) and channel c. After global average pooling, the original feature map of each channel c is simplified to a single value G c , and finally the global average of all channels will form a new feature vector G = (G1, G2, ..., G c ), whose dimension is 1×1×c. This compact feature vector can be directly connected to the classification layer for the final category prediction. H and W represent the height and width of the feature map, respectively.

[0073] Beneficial effects:

[0074] The present invention significantly improves the detection performance of small targets and tail categories by effectively utilizing unlabeled data to strengthen model training. The present invention adopts the Faster R-CNN model combined with the Feature Pyramid Network (FPN) and Global Average Pooling (GAP) technology to realize an end-to-end human ear identification process. The innovation lies in the improvement of the Mean Teacher algorithm process and the introduction of two data enhancement technologies: pixel-level mixed pseudo-label Mixup and image splicing pseudo-label Mosaic. The former reduces the negative impact of missed targets, and the latter enhances the model's recognition ability for small targets by increasing the number of labels for small-scale targets. These technologies, combined with the improved algorithm process, jointly improve the performance of deep convolutional neural networks in semi-supervised target detection tasks.

[0075] The present invention selected Faster R-CNN as the optimal model by conducting all experiments on a high-performance server and comparing the baselines of different models, and made targeted improvements and optimizations to it. Although the generalization ability and robustness of the model need to be further improved, this study provides new ideas for human ear identification technology and has important guiding significance for algorithm improvement and application expansion. In addition, the algorithm framework proposed in the present invention is not only suitable for human ear identification, but can also be extended to other small target detection tasks, and has broad application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0077] Figure 1 It is a schematic diagram of the ear brushing identity recognition method based on pseudo-label semi-supervised learning of the present invention.

[0078] Figure 2 Schematic diagram of the MeanTeacher semi-supervised learning method.

[0079] Figure 3 Schematic diagram of the improved algorithm framework based on MeanTeacher.

[0080] Figure 4 Schematic diagram of the identity recognition algorithm process.

[0081] Figure 5 This is a partial diagram of the EarVN1.0 dataset.

[0082] Figure 6 A partial illustration of the USTB Ear dataset.

[0083] Figure 7 This is a partial diagram of Kaggle's scattered data sets.

[0084] Figure 8 Enhance the pipeline for different data.

[0085] Fig. 9 Schematic diagram of Mixup image aliasing.

[0086] Fig.10 This is a schematic diagram of Mosaic splicing.

[0087] Fig.11a Schematic diagram of the total loss function.

[0088] Fig.11b Schematic diagram of the learning rate change.

[0089] Fig.11c Schematic diagram of the average accuracy of the training set.

[0090] Fig.11d Schematic diagram of the average accuracy of the test set. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] To overcome the above difficulties, increasing the amount of training data has become a solution. The exploration of semi-supervised learning methods aims to alleviate the problem of shortage of high-quality labeled data, thereby improving the performance and generalization ability of ear detection models. Using semi-supervised learning methods, models can be trained with the help of a large amount of unlabeled data on the basis of limited labeled data, effectively enhancing the model's recognition accuracy for new samples.

[0093] like Figure 1 The present invention provides an ear brushing identity recognition method based on pseudo-label semi-supervised learning, comprising:

[0094] S1, construct a data set for training and annotate the data set to obtain a labeled training set, an unlabeled training set, and a labeled validation set;

[0095] S2, determine the convolutional neural network model to be used, and modify its training process and detection process according to the improved algorithm and recognition principle;

[0096] S3, using the improved MeanTeacher algorithm to train the convolutional neural network model to obtain a trained neural network model;

[0097] S4, using the trained neural network model to calculate the human ear image to be identified to obtain a feature vector;

[0098] S5, using the nearest neighbor search algorithm to match the feature vector obtained in S4 with the human ear feature vector that has been entered into the database and needs to be used for identification, to achieve end-to-end human ear identity recognition.

[0099] Preferably, S1 includes:

[0100] Retrieve the public datasets on the Internet, and then fuse the ear images of three different sizes, small, medium, and large, according to a preset ratio (e.g., 9:4:1), to construct a dataset;

[0101] For example, if the number of horizontal or vertical pixels is less than or equal to 512, it is a small size; if the number of horizontal or vertical pixels is greater than 512 and less than or equal to 1280, it is a medium size;

[0102] All sizes greater than 1280 are large;

[0103] Through manual review, we remove erroneous images, duplicate images, and images that are too difficult to identify in the constructed dataset; erroneous images include images that contain ears but are not human ears;

[0104] The processed data set is divided into three parts: a labeled training set, an unlabeled training set, and a labeled validation set; the number of images in the labeled training set is 1% of the total number of ear images in the data set;

[0105] LabelImg software is used to annotate the data of the labeled training set and the labeled validation set. The human ear area in the image is framed with a target box, and the framed area is labeled with the ear label.

[0106] The following is a detailed description of the data sources used in each part:

[0107] EarVN1.0: This part occupies the largest proportion of the total dataset. It is constructed by collecting ear images of 164 Asians by Ho Chi Minh Open University in Vietnam in 2018. It includes a total of 28,412 color images, including 98 males and 66 females. Its unique value lies in the fact that the camera system is used in the image acquisition stage to record the facial information of individuals under various lighting conditions, covering a wide range of postures, scales and lighting changes. Some diagrams of the EarVN1.0 dataset are shown below: Figure 5 .

[0108] USTB Ear: This dataset was taken by the Notre Dame University team and contains 2D and 3D ear images. After filtering out the 3D part and removing duplicate images due to orientation differences, a total of 308 color ear images of 77 teachers and students were integrated. Some diagrams of the USTBEar dataset are shown below: Figure 6 .

[0109] Kaggle scattered data sets: There are many public ear data sets related to competitions on the Kaggle platform. Although some data sets are used for ear detection or ear acupoint recognition, their actual sources and contents are difficult to verify because some uploaders did not provide detailed descriptions. After self-collection and elimination of materials that do not meet research needs, a total of nearly 6,000 color and black-and-white ear images were collected. Some of the Kaggle scattered data sets are shown below. Figure 7 .

[0110] There is no labeling information in the self-built dataset, and even if there is, it is not ear labeling information, so the data needs to be processed. Through manual review, the wrong images, duplicate images, and images that are too difficult to identify in the self-built dataset were removed.

[0111] The final comprehensive dataset contains 33,591 ear images, covering small, medium, and large sizes, as well as different scenarios where the ear occupies a small or large area in the image. Among them, "size" refers to the overall size of the image, and "object size" specifically refers to the relative area ratio of the ear in the image.

[0112] A script was written to divide the above dataset into three parts: "labeled training set", "unlabeled training set" and "labeled validation set", ensuring that the image size and target size distribution of each category are balanced, with the number of images being 455, 30,270 and 2,866 respectively. In this way, the labeled training samples account for about 1% of the total dataset. Subsequently, LabelImg software was used to annotate the data of the labeled training set and the labeled validation set.

[0113] Preferably, the image with too high recognition difficulty is an image without ear contour.

[0114] Preferably, the convolutional neural network model includes a Faster-RCNN model.

[0115] Preferably, S3 includes:

[0116] In the first iteration, the following operations are performed:

[0117] Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b;

[0118] Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image;

[0119] Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence to generate a pseudo label data set; store the pseudo label data set in the cache library; low confidence refers to a confidence level less than 0.85;

[0120] Starting from the second iteration, the following operations are performed:

[0121] Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b;

[0122] Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image;

[0123] Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence and generate a pseudo label data set; use A to represent the pseudo label data set obtained in the current iteration, and store A in the cache library; low confidence refers to a confidence level less than 0.85;

[0124] Extract multiple images from the cache library to form a set B, where the set B does not contain the images in the set A, and the number of images in the set B is the same as the number of images in the set A;

[0125] Mix the images in set A and set B by Mixup to obtain a first image set;

[0126] Processing set A and images extracted from set B by Mosaic to obtain a second image set; the number of images extracted from set B is the same as the number of images in set A;

[0127] Performing strong enhancement processing on the images in the first image set and the second image set respectively to obtain a third image set;

[0128] Performing labeled supervised enhancement processing on the labeled image to obtain an enhanced image;

[0129] Mixing the enhanced image and the third image set as input data of the student model;

[0130] Through the supervision loss L supervised and the unsupervised loss L unsupervised Optimize the student model;

[0131] The exponential moving average strategy is used to update the teacher model.

[0132] Meanteacher is a very famous semi-supervised learning algorithm, such as Figure 2 As shown in the figure, MeanTeacher contains two models: a teacher model and a student model. These two models have the same weights and the same structure at the beginning.

[0133] In each training iteration, the system randomly samples labeled images and unlabeled images to form a batch of data. For each unlabeled image, two views are generated through weak augmentation and strong augmentation.

[0134] The teacher model predicts the weakly enhanced unlabeled images and generates pseudo labels. The confidence threshold is used to filter out the low-confidence pseudo labels, and the remaining pseudo labels are converted to the strongly enhanced view to form a pseudo-labeled dataset.

[0135] The student model is updated by optimizing the supervised loss of labeled data and the unsupervised loss of pseudo-labeled data. The weight of the teacher model will be frozen, but as the student model changes, the teacher model must also improve accordingly. After the student model surpasses the teacher model, the teacher model can update its own parameters based on the weight of the student model through the exponential moving average (EMA), that is, the knowledge learned by the student model can be transferred to the teacher model, and the parameters of the teacher model will only change at this time.

[0136] The hybrid pseudo-labeling algorithm proposed in this paper is based on MeanTeacher and improved, and also adopts the teacher-student model architecture. This method aims to improve the quality and quantity of pseudo-labels by integrating multiple data enhancement techniques, thereby enhancing the performance of the model in semi-supervised object detection tasks.

[0137] For the overall process, for example, if an iteration contains one labeled image and four unlabeled images, the algorithm also selects another four pseudo-labeled images from the cache, and obtains four mixed images through pseudo-label Mixup processing; then, eight of these images (including the enhanced version of the original four unlabeled images) are randomly selected to generate a new mixed image through pseudo-label Masaic technology. Finally, the five images generated by this series of operations replace the original four unlabeled images as the training input of the student model.

[0138] Through this systematic strategy, the proposed algorithm effectively improves the performance of multiple types of target detectors (including one-stage, two-stage, end-to-end, etc.) in semi-supervised environments, especially small target detection.

[0139] The present invention improves the generalization ability of the model by increasing the amount of data. To ensure that the proposed solution has high feasibility and verifiability, a data enhancement strategy is adopted to enrich the data samples. In view of space efficiency, data enhancement is only applied in real time during actual runtime without static storage. To further improve the reproducibility of the experiment, a random processing mechanism is introduced, which specifically includes the definition of two key modules: color_space and Geometric.

[0140] color_space covers a variety of color transformation operations, such as auto contrast, equalization, sharpening, toning, solarization, color adjustment, contrast adjustment, and brightness adjustment.

[0141] geometric includes a series of geometric transformation operations, including rotation, X-axis shearing, Y-axis shearing, X-axis translation, and Y-axis translation.

[0142] In actual operation, depending on whether the data is labeled, this study designed different data enhancement processes for different data types, such as Figure 8 shown.

[0143] Preferably, the weak enhancement process includes:

[0144] The first step is to load the image to be weakly enhanced;

[0145] In the second step, the image in the first step is randomly scaled;

[0146] The third step is to randomly flip the image obtained in the second step.

[0147] Preferably, the process of label-supervised enhancement processing includes:

[0148] The first step is to load the image and the corresponding annotation information;

[0149] In the second step, the image in the first step is randomly scaled;

[0150] The third step is to randomly flip the image obtained in the second step;

[0151] The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step;

[0152] The fifth step is to filter out target frames that are too small and the images where the target frames are located. Target frames that are too small are target frames in images with a size less than 80×80.

[0153] Preferably, the process of strong enhancement processing includes:

[0154] The first step is to load the image to be weakly enhanced;

[0155] In the second step, the image in the first step is randomly scaled;

[0156] The third step is to randomly flip the image obtained in the second step;

[0157] The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step;

[0158] Step 5: Apply a geometric-based random enhancement strategy to the image obtained in step 4.

[0159] Step 6: Apply a random enhancement strategy based on color_space to the image obtained in step 5;

[0160] Step 7: Apply a geometric-based random enhancement strategy to the image obtained in step 6;

[0161] In the eighth step, the target frames that are too small and the images where the target frames are located are filtered out. The target frames that are too small are the target frames in the images whose size is less than 80×80.

[0162] The strong_pipeline is used for the student model of unlabeled data. Compared with the teacher model, the strong_pipeline is more complex and adds a geometric transformation step. This design is intended to provide more challenging training samples for the student model within the semi-supervised learning framework to enhance its robustness to geometric changes.

[0163] In addition to the above data enhancement process, the present invention introduces two additional data enhancement methods.

[0164] The first is pseudo-label Mixup data enhancement. In the field of CV, Mixup is a data enhancement method that operates on images at the pixel level, mainly used in image classification tasks. In order to reduce the negative impact of missed detections in pseudo-labels on model training and enhance the gradient response of correct pseudo-labels, this paper uses Mixup data enhancement in the strong enhancement process of unlabeled data, named pseudo-label Mixup.

[0165] The specific workflow is:

[0166] (1) Image aliasing: Select two images I with pseudo-annotations. i and I j , and their corresponding pseudo-annotation set B i and Bj , where each set contains multiple bounding boxes and their category labels. Apply linear interpolation to alias the image and generate a mixed image I mix ,like Fig. 9 As shown, the specific formula is:

[0167] I mix =λ·I i +(1-λ)·I j

[0168] Among them, λ is a random weight sampled from the Beta distribution Beta(α,α), and α is a hyperparameter.

[0169] (2) Pseudo-label merging: For I mix For each pixel position in , its corresponding pseudo annotation is represented by I i and I j The pseudo-annotations at the corresponding positions in are linearly merged according to the weight of λ. Specifically, if the position is covered by bounding boxes in both images, the intersection of the two is taken; if only one image is covered by a bounding box, the box is directly used; if there is no bounding box in both images, it is considered that there is no annotation at this position. In this way, the mixed image I is obtained mix The pseudo-annotation set

[0170] The second is pseudo-label Mosaic data enhancement. Mosaic data enhancement is also commonly used in image classification tasks. It simulates a larger image size by splicing multiple images together to increase the contextual information of the image. In order to improve the model's detection ability for small-sized targets, this paper uses the Mosaic data enhancement algorithm in the strong enhancement process of unlabeled data, and increases the number of small and medium-sized target labels by downsampling and splicing images. It is named pseudo-label Mosaic. The specific workflow is:

[0171] Image scaling: Select four images I1, I2, I3, I4 with pseudo annotations, and their corresponding pseudo annotation sets Since the task only needs to determine whether an object exists without paying attention to its specific location, this paper uses the maximum pooling operation to scale down these four images proportionally (e.g., 1 / 2 or 1 / 4) to obtain the scaled-down images I′1, I′2, I′3, and I′4. Since downsampling reduces the size of the image, objects that may have been considered large targets in the original image may become small or medium-sized targets in the downsampled image. In this way, when these downsampled images are spliced ​​into a large image, the new image will contain pseudo-labels of small and medium-sized targets of more sizes.

[0172] (2) Image stitching: The specific effects are as follows: Fig.10, concatenate the reduced images into a single composite image I in a 2x2 manner mosaic :

[0173]

[0174] (3) Pseudo-annotation conversion: For the composite image I mosaic For each pixel position in , the corresponding pseudo-annotation is migrated from the pseudo-annotations of the corresponding positions in the original four images. If a position is covered by bounding boxes in multiple small images, these boxes are merged and the category label is updated to the category that appears most frequently in all covered boxes to reflect the possible overlap of multiple categories.

[0175] Preferably, for supervised learning, a smooth L1 loss function is used:

[0176]

[0177] L1(x,y) represents the smooth loss function, x and y represent the predicted value and true value of the neural network model respectively;

[0178] For unsupervised learning, use the consistency regularization loss:

[0179]

[0180] N represents the number of samples in each batch; L consistency represents the consistency regularization loss function; f(x i ) represents the output result of the student model for the i-th image; g(x i ) represents the output result of the teacher model calculating the i-th image;

[0181] In each training iteration, the parameters of the student model are updated by minimizing the total loss function, which is the weighted sum of these two losses, L total is the total loss function, L supervised To monitor the loss, L unsupervised is the unsupervised loss, weight w u Used to balance the contributions between the two:

[0182] L total =L supervised +w u *L unsupervised .

[0183] Preferably, S2 includes:

[0184] The convolutional neural network model to be used is the Faster R-CNN model;

[0185] Modify the FPN and global average pooling layers in the neck part of the Faster-RCNN model to enable the Faster-RCNN model to output the feature vector of the ear in the human ear image to be identified, and then train the modified model;

[0186] Inputting the human ear image to be identified into the trained model, and obtaining the feature vector of the ear in the human ear image output by the trained model;

[0187] The global average pooling operation is calculated as follows:

[0188]

[0189] Among them, G c represents the output value of the cth channel after global average pooling; F i,j,c Represents the pixel value of the original feature map at position (i, j) and channel c. After global average pooling, the original feature map of each channel c is simplified to a single value G c , and finally the global average of all channels will form a new feature vector G = (G1, G2, ..., G c ), whose dimension is 1×1×c. This compact feature vector can be directly connected to the classification layer for the final category prediction. H and W represent the height and width of the feature map, respectively.

[0190] Experimental verification and analysis:

[0191] The model evaluation indicators used are all very commonly used evaluation indicators in target detection tasks in deep learning, as follows:

[0192] Average Precision (AP). For a specific category, we need to get all the predicted bounding boxes of that category and their corresponding confidence scores, as well as the corresponding ground truth bounding boxes. First, we sort the predicted bounding boxes from high to low according to the confidence scores, and then calculate the AP by the following steps:

[0193] Precision: For the current confidence threshold, calculate the ratio of the number of correctly detected samples (TruePositives, TP) to the total number of detection results (TP + FalsePositives, FP). TP means that the intersection over union (IoU) of the predicted bounding box and any true annotated bounding box is greater than the predefined threshold, and the true annotated bounding box has not been matched by other predicted boxes with higher confidence; FalsePositives means that the predicted bounding box fails to meet the above conditions.

[0194] Recall: For the current confidence threshold, calculate the ratio of the number of correctly detected samples (TP) to the total number of actual positive samples (TP + False Negatives, FN). FN refers to the part of all GT that has not been correctly matched by any prediction box.

[0195] Precision-Recall Curve (PR curve): As the confidence threshold gradually decreases, we will get a series of Precision and Recall values. The curve plotting the change of Precision with Recall is the PR curve.

[0196] AP: For each Recall value on the PR curve, calculate the corresponding maximum Precision, and then average these maximum Precision values ​​to get AP. The calculation rule is expressed as the following formula:

[0197]

[0198] Among them, m is the number of equally spaced recall points (such as the commonly used 101 points), P interp (r i ) is the recall rate r i The interpolation accuracy value at .

[0199] Mean Average Precision (mAP). Due to different IoUs, mAP is the average of the average precisions of multiple different rules. It is used to measure the detection accuracy of the model on different categories. The average is used to get the overall detection performance. The calculation formula is as follows:

[0200]

[0201] Where N is the total number of categories, AP c Represents the Average Precision of category c.

[0202] The intersection-over-union ratio is used to measure the degree of overlap between the predicted bounding box and the true annotated bounding box. The specific calculation formula is as follows:

[0203] Given a predicted bounding box PbB and a true labeled bounding box GtB, they are respectively represented by the upper left corner coordinates and the lower right corner coordinates definition.

[0204] Intersection Area: The area of ​​the intersection of two bounding boxes, denoted as I(A), is calculated as:

[0205]

[0206] Union Area: The total area of ​​two bounding boxes, including the intersecting parts and the unique parts of each, is recorded as U(A) and is calculated as:

[0207]

[0208] IoU: The ratio of the intersection area to the union area, that is:

[0209]

[0210] The value range of IoU is [0,1]. IoU=0 means that the two bounding boxes have no intersection, IoU=1 means that the two bounding boxes completely overlap, and the value between the two is 0 to 1.

[0211] After repeated training and parameter adjustment, this paper finally determined the following hyperparameter settings for the experiment: AdamW was selected as the optimizer, its basic learning rate was set to 0.0002, and the regularization parameter of weight decay was set to 0.0001. For the learning rate scheduling strategy, this paper designed three stages:

[0212] Linear warm-up phase: The initial learning rate increases linearly from 0 to the base learning rate of 0.0002, and it takes 5000 iterations to complete this process.

[0213] Static hold phase: The learning rate is kept constant at 0.0002 until the 35000th iteration.

[0214] Cosine annealing phase: From this point on, the cosine annealing strategy (CosineAnnealingLR) is used to adjust the learning rate. In the next 40,000 iterations, this strategy gradually decays the learning rate from the current value to zero, following the decay law of the cosine function.

[0215] The cosine annealing learning rate is updated as follows:

[0216]

[0217] Among them, η t is the learning rate in the current training cycle (e.g., iteration t); η min and η max are the minimum learning rate and the initial maximum learning rate respectively; T max is the total number of training cycles; T cur is the current training cycle.

[0218] To constrain the magnitude of the gradient update, this paper applies gradient clipping, sets the maximum norm of the gradient update to 0.1, and uses the L2 norm for calculation. In addition, the batch size (batch_size) is set to 8, and the number of planned iterations for the entire training process is 90,000. The data loader uses the custom loading process mentioned above.

[0219] According to the above data, Faster R-CNN was finally selected as the final model. It was modified based on the original baseline model, adding the GAP integration and output vectorization processing described above, and performing end-to-end model training. The results are shown in Figure 11.

[0220] According to the curve chart in TensorBoard, the final loss function trend tends to be stable. When evaluating the validation set at the 87,500th iteration, the highest average accuracy reached 82.7% (IoU used for mAP is 0.5 to 0.95). On the labeled training set, the average accuracy is as high as 98.4%, and the difference between the training set accuracy and the validation set accuracy is 15.7%, indicating that the model may be at risk of overfitting. The final effect did not achieve a very satisfactory result, but it also provided a clear path for future optimization: the model already has a certain feature extraction capability, but in order to ensure its effectiveness in practical applications, measures need to be taken to improve its generalization ability, such as reducing overfitting, performing hyperparameter tuning, and using more data for training.

[0221] Fig.11a Schematic diagram of the total loss function. Fig.11b Schematic diagram of the learning rate change. Fig.11c Schematic diagram of the average accuracy of the training set. Fig.11d Schematic diagram of the average accuracy of the test set.

[0222] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An ear brushing identity recognition method based on pseudo-label semi-supervised learning is characterized by: include: S1, construct a data set for training and annotate the data set to obtain a labeled training set, an unlabeled training set, and a labeled validation set; S2, determine the convolutional neural network model to be used, and modify its training process and detection process according to the improved algorithm and recognition principle; S3, using the improved MeanTeacher algorithm to train the convolutional neural network model to obtain a trained neural network model; S4, using the trained neural network model to calculate the human ear image to be identified to obtain a feature vector; S5, using the nearest neighbor search algorithm to match the feature vector obtained in S4 with the human ear feature vector that has been entered into the database and needs to be used for identification, to achieve end-to-end human ear identity recognition.

2. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 1 is characterized in that: S1 includes: Retrieve public datasets on the Internet, and then fuse ear images of three different sizes, small, medium, and large, according to a preset ratio to construct a dataset; Through manual review, we remove erroneous images, duplicate images, and images that are too difficult to identify in the constructed dataset; erroneous images include images that contain ears but are not human ears; The processed data set is divided into three parts: a labeled training set, an unlabeled training set, and a labeled validation set; the number of images in the labeled training set is 1% of the total number of ear images in the data set; The LabelImg software is used to annotate the data of the labeled training set and the labeled validation set. The human ear area in the image is framed with a target box, and the framed area is labeled with the ear label.

3. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 2 is characterized in that: An image in which the ear contour does not exist is an image in which the recognition difficulty is too high.

4. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 1 is characterized in that: Convolutional neural network models include the Faster-RCNN model.

5. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 1 is characterized in that S3 include: In the first iteration, the following operations are performed: Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b; Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image; Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence to generate a pseudo label data set; store the pseudo label data set in the cache library; low confidence refers to a confidence level less than 0.85; Starting from the second iteration, the following operations are performed: Images are extracted from the labeled training set and the unlabeled training set in a ratio of 1:4 as the images of the current training batch. b represents the number of labeled images, and the number of unlabeled images is 4b; Perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image; Use the teacher model to predict the weakly enhanced image to obtain the pseudo label of the weakly enhanced image, thereby obtaining the pseudo label image; filter out the pseudo label images with low confidence and generate a pseudo label data set; use A to represent the pseudo label data set obtained in the current iteration, and store A in the cache library; low confidence refers to a confidence level less than 0.85; Extract multiple images from the cache library to form a set B, where the set B does not contain the images in the set A, and the number of images in the set B is the same as the number of images in the set A; Mix the images in set A and set B by Mixup to obtain a first image set; Processing set A and images extracted from set B by Mosaic to obtain a second image set; the number of images extracted from set B is the same as the number of images in set A; Performing strong enhancement processing on the images in the first image set and the second image set respectively to obtain a third image set; Performing labeled supervised enhancement processing on the labeled image to obtain an enhanced image; Mixing the enhanced image and the third image set as input data of the student model; Through the supervision loss L supervised and the unsupervised loss L unsupervised Optimize the student model; The exponential moving average strategy is used to update the teacher model.

6. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 5 is characterized in that: The weak enhancement process includes: The first step is to load the image to be weakly enhanced; In the second step, the image in the first step is randomly scaled; The third step is to randomly flip the image obtained in the second step.

7. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 5 is characterized in that: The process of label-supervised enhancement includes: The first step is to load the image and the corresponding annotation information; In the second step, the image in the first step is randomly scaled; The third step is to randomly flip the image obtained in the second step; The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step; The fifth step is to filter out target frames that are too small and the images where the target frames are located. Target frames that are too small are target frames in images with a size less than 80×80.

8. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 5 is characterized in that: The process of strong enhancement includes: The first step is to load the image to be weakly enhanced; In the second step, the image in the first step is randomly scaled; The third step is to randomly flip the image obtained in the second step; The fourth step is to apply a random enhancement strategy based on color_space to the image obtained in the third step; Step 5: Apply a geometric-based random enhancement strategy to the image obtained in step 4. Step 6: Apply a random enhancement strategy based on color_space to the image obtained in step 5; Step 7: Apply a geometric-based random enhancement strategy to the image obtained in step 6; In the eighth step, the target frames that are too small and the images where the target frames are located are filtered out. The target frames that are too small are the target frames in the images whose size is less than 80×80.

9. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 5 is characterized in that: For supervised learning, a smooth L1 loss function is used: L1(x,y) represents the smooth loss function, x and y represent the predicted value and true value of the neural network model respectively; For unsupervised learning, use the consistency regularization loss: N represents the number of samples in each batch; L consistency represents the consistency regularization loss function; f(x i ) represents the output result of the student model for the i-th image; g(x i ) represents the output result of the teacher model calculating the i-th image; In each training iteration, the parameters of the student model are updated by minimizing the total loss function, which is the weighted sum of these two losses, L total is the total loss function, L supervised To monitor the loss, L unsupervised is the unsupervised loss, weight w u Used to balance the contributions between the two: L total =L supervised +w u *L unsupervised 。 10. The ear brushing identity recognition method based on pseudo-label semi-supervised learning according to claim 4, characterized in that S2 include: The convolutional neural network model to be used is the Faster R-CNN model; Modify the FPN and global average pooling layers in the neck part of the Faster-RCNN model to enable the Faster-RCNN model to output the feature vector of the ear in the human ear image to be identified, and then train the modified model; Inputting the human ear image to be identified into the trained model, and obtaining the feature vector of the ear in the human ear image output by the trained model; The global average pooling operation is calculated as follows: Among them, G c represents the output value of the cth channel after global average pooling; F i,j,c Represents the pixel value of the original feature map at position (i, j) and channel c. After global average pooling, the original feature map of each channel c is simplified to a single value G c , and finally the global average of all channels will form a new feature vector G = (G1, G2, ..., G c ), whose dimension is 1×1×c. This compact feature vector can be directly connected to the classification layer for the final category prediction. H and W represent the height and width of the feature map, respectively.

Images