A small target detection method based on semi-supervised and feature fusion

Abstract

The application discloses a small target detection method based on semi-supervision and feature fusion, and steps of the method comprise the following steps: 1, constructing a deep learning network framework of a target detection model; 2, preparing an original image dataset and a semi-supervised dataset; 3, constructing a pseudo label framework under semi-supervised learning; 4, adding a novel feature fusion module RE_BiFPN; 5, training the network and obtaining a novel detection model. The application can obtain more abundant and more accurate image features through the features extracted by a convolutional neural network and the feature pyramid, and can obtain more abundant semantic information according to high-level and low-level feature fusion, so that small target faces can be detected with high accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a face image retrieval method based on semi-supervised object detection. Background Technology

[0002] Large-scale datasets and computational resources enable deep neural learning networks to exhibit superior performance across various tasks. However, training supervised deep learning networks requires a large number of labeled samples, which are very costly to label. As an alternative to supervised learning, semi-supervised learning methods have gained increasing attention in recent years, but these are mostly geared towards image classification tasks. Applications of semi-supervised learning in object detection are still relatively limited, and exploration of its application in face detection is even scarcer. Summary of the Invention

[0003] This invention aims to address the shortcomings of existing technologies by proposing a small target detection method based on semi-supervised and feature fusion, which seeks to acquire richer feature information from face images, thereby improving the detection accuracy and speed of small target faces in large scenes.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The small target detection method based on semi-supervised and feature fusion of this invention is characterized by the following steps:

[0006] Step 1: Construct a face image dataset and perform preprocessing;

[0007] Step 1.1: Obtain a labeled dataset of real face images, perform size normalization and preprocessing to obtain a preprocessed face image dataset of dimension M×M×C, denoted as X={x1,x2,…,x…}. i ,…,x n}, where x i Let the i-th face image be represented by the label set T = {t1, t2, ..., t3}. i ,…t n}, where t i Let x represent the i-th face image. i The corresponding labels are i = 1, 2, ..., n, where n is the total number of images in the labeled face image dataset; M represents the length and width of the processed image, and C represents the number of channels;

[0008] Step 1.2: Obtain the unlabeled face image dataset, perform size normalization and preprocessing, and obtain the preprocessed face image dataset with dimensions M×M×C, denoted as Y={y1,y2,…,y j ,…ym}, where y j Let j represent the j-th unlabeled face image, where j = 1, 2, ..., m, and m is the total number of images in the unlabeled face image dataset.

[0009] Step 2: Construct a semi-supervised learning-based object detection network, including: backbone network, feature pyramid (FPN), SSH module, head module, and prediction module;

[0010] Step 2.1: Construct the backbone network and perform three-stage feature extraction;

[0011] The backbone network contains three-stage separable convolutional conv_dw modules; the kernel size of the convolutional layer in each separable convolutional conv_dw module is a×a, and the stride is k;

[0012] The labeled i-th face image x i and its corresponding tag t i After encoding, the input feature S with dimensions M×M×C is obtained. i ,S i After processing by the first-stage separable convolution conv_dw module, the first-stage network features S are obtained with dimensions of (M / p)×(M / p)×(C×p). i_p S i_p After further processing by the second-stage separable convolution conv_dw module, the second-stage network features S with dimensions of (M / q)×(M / q)×(C×q) are obtained. i_q Finally, Si _q After processing by the third-stage separable convolution conv_dw module, the network feature S of the third stage with a dimension of (M / r)×(M / r)×(C×r) is obtained. i_r Where p represents the feature compression ratio of the first stage, q represents the feature compression ratio of the second stage, and r represents the feature compression ratio of the third stage;

[0013] Step 2.2: Construct the Feature Pyramid (FPN) and extract image features;

[0014] The Feature Pyramid Network (FPN) utilizes b×b ​​convolution pairs. and After adjusting the number of channels, upsampling is then performed on the three adjusted features to obtain the first upsampled feature. Second upsampling feature and third upsampling features Will and The first fusion feature S is obtained after fusion. i_qr ,Will and After fusion, the second fusion feature S i_pq is obtained

[0015] Step 2.3: strengthening feature extraction through an SSH module

[0016] The SSH module performs a splicing operation on the features S i_pq , S i_qr and S i_pqr to obtain a spliced feature S i_pqr ;

[0017] The SSH module processes the spliced feature S i_pqr using three parallel convolutions with kernel sizes of dxd, exd and fxf, respectively, to obtain a first-stage strengthened feature S i_pout , a second-stage strengthened feature S i_qout and a third-stage strengthened feature S i_rout ;

[0018] Step 2.4: constructing a Head module and extracting features of classification, prediction frame and key points

[0019] The Head module combines S i_pout , S i_qout and S i_rout into a prediction feature S i_new , and then performs tensor reconstruction of three dimensions on S i_new , respectively, to form a face class feature S i_class with a channel number of U, a face frame feature S i_box with a channel number of V and a face key point feature S i_lands with a channel number of W

[0020] Step 2.5: establishing a loss function

[0021] The total loss function L i of the i-th face image x i is constructed using formula (1)

[0022] L i = w i_class ×L i_class + w i_box ×L i_box + w i_lands ×L i_lands (1)

[0023] In formula (1), L i_class represents the average absolute and mean square error loss of the face class of the i-th face image x i , L i_boxLet x represent the i-th face image. i Mean absolute and mean squared error loss of face bounding boxes, L i_lands Let x represent the i-th face image. i Mean absolute and mean squared error loss of facial landmarks, w i_class Let x represent the i-th face image. i Face category loss weights, w i_box Let x represent the i-th face image. i Face bounding box loss weights, w i_lands Let x represent the i-th face image. i Loss weights for facial landmarks;

[0024] Step 2.6: Correction and decoding of the prediction module;

[0025] The prediction module will S i_class ,S i_box and S i_lands After concatenation, the final predicted feature S with (U+V+W) channels is obtained. i_out Then for S i_out After decoding, the i-th face image x is obtained. i The prediction results include face bounding boxes and facial landmarks;

[0026] Non-maximum suppression is used to remove prediction boxes with high overlap in the prediction results, thus obtaining the i-th face image x. i The final prediction result f i Let the final prediction result f i The position of any face bounding box in the image is denoted as (x). i ,y i ,w i ,h i ), where (x i ,y i ) represents the coordinates of the center point of the face bounding box, w i ,h i The width and height of the corresponding face bounding box; let the prediction result f i The set of facial key points in the image is {(x ih ,y ih )|h=1,2,…,H}, where x ie ,y ie These represent the h-th key point of the face, where H represents the number of key points;

[0027] Step 3: Based on the face image dataset X, train the object detection network using gradient descent and calculate the total loss function L. i To update the network parameters until the total loss function L is reached. iConverge to stop, so as to obtain the trained target detection network and as a teacher model G;

[0028] Step 4: Constructing pseudo-labels of the semi-supervised learning dataset;

[0029] Inputting the unlabeled face image dataset Y into the teacher model G for processing to obtain the pseudo-labels F of Y Y ={fy1, fy2…fy j …fy m}; wherein, fy j represents the corresponding predicted label of the unlabeled image data y j .

[0030] Mixing the labeled face image dataset X and the pseudo-labeled face image dataset Y as a new face image dataset XY={xy1, xy2…xy g …xy m+n}, wherein, xy g represents the gth new face image, and let the new face image dataset XY correspond to a new label set TF={tf1, tf2…tf g …tf m+n}, wherein, tf g represents the label of the gth new face image xy g ; g=1, 2, 3…m+n, and m+n is the total number of images of the new face image dataset XY;

[0031] Step 5: Constructing and training a new semi-supervised detection network, wherein the new semi-supervised detection network is composed of the backbone network in the teacher model G, the SSH module, the prediction module and the new pyramid module RE_BiFPN;

[0032] Step 5.1: Three-stage feature extraction based on the backbone network;

[0033] Inputting the gth new face image xy g and the corresponding label tf g into the backbone network of the teacher model G for processing to obtain a new first-stage network feature S g_p , a new second-stage network feature S g_p and a new third-stage network feature S g_r ;

[0034] Step 5.2: Constructing the new pyramid module RE_BiFPN and performing image feature extraction;

[0035] Step 5.2.1: The new pyramid module RE_BiFPN utilizes a convolution with a dimension of b×b to process S g_p , S g_p and Sg_r After adjusting the number of channels of each of the three features respectively, adjusted features and are obtained

[0036] Step 5.2.2, the three fusion features are respectively fused with each other, and then fused to obtain three stage fusion features

[0037] Step 5.3: enhanced feature extraction is performed through an SSH module;

[0038] The SSH module in the teacher model G performs splicing operation on the three stage fusion features and to obtain splicing features S g_pqr ;

[0039] The SSH module processes the splicing features S g_pqr using parallel convolution with three convolution kernel sizes dxd, exd and fxd, and correspondingly obtains new first stage enhanced features S g_pout , new second stage enhanced features S g_qout and new third stage enhanced features S g_rout ;

[0040] Step 5.4: features of classification, prediction frame and key point are extracted based on a Head module;

[0041] The Head module combines S g_pout , S g_qout and S g_rout into new prediction features S g_new , and then performs tensor reconstruction operation of three dimensions on S g_new respectively to obtain new face class features S g_class , new face frame features S g_box and new face key point features S g_lands ;

[0042] Step 5.5: a total loss function L new of the new semi-supervised detection network is established using formula (2):

[0043] L new＝ αL i +βL g (2)

[0044] In formula (2), L g represents the new semi-supervised detection network for the gth new face image xy gThe loss function of the teacher model G, alpha represents the weight of the labeled image supervision loss, and beta represents the weight of the pseudo-labeled image supervision loss.

[0045] Step 5.6: Correction and decoding of the prediction module;

[0046] The prediction module in the teacher model G will S g_class , S g_box and S g_lands are spliced to obtain a new final prediction feature S g_out , and after decoding S g_out , a new gth face image xy g is obtained.

[0047] Step 6: Based on the face image dataset XY, the gradient descent method is used to train the new semi-supervised detection network, and the total loss function L new is calculated to update the network parameters until the total loss function L new converges, thereby obtaining the trained target detection network as a new detection model G new .

[0048] The small target detection method based on semi-supervision and feature fusion has the characteristics that,

[0049] The two-by-two fusion process in step 5.2.1 includes:

[0050] After the features are subjected to separable depth convolution conv, the first layer feature of the third stage is obtained.

[0051] The features and features are fused by using a double-layer feature fusion formula, and the first layer feature of the second stage is obtained.

[0052] The features and features are fused by using a double-layer feature fusion formula, and the first layer feature of the first stage is obtained.

[0053] The features and features are fused by using a double-layer feature fusion formula, and the second layer feature of the first stage is obtained.

[0054] The features and features are fused by using a double-layer feature fusion formula, and the third layer feature of the first stage is obtained.

[0055] fuse the features and , and obtain the first layer features of the second stage

[0056] fuse the features and , and obtain the second layer features of the second stage

[0057] fuse the features and , and obtain the third layer features of the second stage

[0058] fuse the features and , and obtain the second layer features of the third stage

[0059] fuse the features and , and obtain the third layer features of the third stage

[0060] The two-by-two fusion process in step 5.2.2 is to take as the adjusted features and and process them according to the fusion manner of step 5.2.1 to obtain the three-stage fused features

[0061] The double-layer feature fusion formula is:

[0062]

[0063] In formula (3), S one ,S two are two input features, w one ,w two are learning parameters corresponding to S one ,S two , Resize represents a down-sampling operation; and ε is a bias hyperparameter.

[0064] The three-layer feature fusion formula is:

[0065]

[0066] In formula (4), S​one two three are three input features, w one two three are S one two three corresponding learning parameters, Resize represents a down-sampling operation; and epsilon is a bias hyper-parameter.

[0067] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the small target detection method, and the processor is configured to execute the program stored in the memory.

[0068] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any small target detection method are executed.

[0069] Compared with the prior art, the present application has the following beneficial effects:

[0070] 1. The present application is not only limited to the modification of the network framework in the target detection framework, but also focuses on the processing of data, uses more unlabeled data, and detects in the currently optimal detection algorithm, so that a more optimal detection precision is obtained.

[0071] 2. The present application extracts feature information at different positions of the face, fuses features of different dimensions and depths, extracts more accurate and rich image features, judges whether the image contains hidden information according to the features, and reorganizes the hidden information, so as to extract more effective features and improve the precision of face recognition.

[0072] 3. The present application proposes a new FPN feature fusion structure RE_BiFPN, which fuses multi-dimensional network features and fully extracts hidden information, and achieves more excellent results in face target detection.

[0073] 5. In the present application, by using the semi-supervised learning method, the available data set of target detection is expanded, a large amount of unlabeled data set is expanded, and the available image resources are fully utilized, so that the face detection target effect of different sizes is more optimal. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is the overall flowchart of the small target detection method in the present application;

[0075] Figure 2 is the structure diagram of the new feature fusion method based on semi-supervised target detection in the present application.​​​​​​ Detailed Implementation

[0076] In this embodiment, a small target detection method based on semi-supervised and feature fusion is presented. This method primarily utilizes a deep multi-dimensional feature extraction method to fuse richer and more accurate features, building upon the large dataset generated by a semi-supervised network. Through training, the neural network can extract detailed feature information from the target image, which is then used for target image retrieval, significantly improving the accuracy of small target face detection in large scenes. For example... Figure 1 As shown, this small target face detection method is performed according to the following steps:

[0077] Step 1: Construct a face image dataset and perform preprocessing;

[0078] Step 1.1: Obtain a labeled dataset of real face images, perform size normalization and preprocessing to obtain a preprocessed face image dataset of dimension M×M×C, denoted as X={x1,x2,…,x…}. i ,…,x n}, where x i Let the i-th face image be represented by the label set T = {t1, t2, ..., t3}. i ,…t n}, where t i Let x represent the i-th face image. i The corresponding labels are i = 1, 2, ..., n, where n is the total number of images in the labeled face image dataset; M represents the length and width of the processed image, and C represents the number of channels;

[0079] Step 1.2: Obtain the unlabeled face image dataset, perform size normalization and preprocessing, and obtain the preprocessed face image dataset with dimensions M×M×C, denoted as Y={y1,y2,…,y j ,…y m}, where y j Let j represent the j-th unlabeled face image, where j = 1, 2, ..., m, and m is the total number of images in the unlabeled face image dataset.

[0080] This invention uses publicly available face data (WiderFace) as labeled data and selects face-related images from the COCO dataset as unlabeled data for semi-supervised experiments. Both face datasets contain data from various aspects of daily life and work, making them more universal and resulting in more robust trained models.

[0081] The WIDERFACE dataset adopted in the present application is a benchmark dataset for face detection, containing 32203 images and 393,703 labeled faces, of which 158,989 labeled faces are in the training set and 39,496 are in the verification set. Each subset contains three levels of detection difficulty: Easy, Medium, and Hard. These faces have a wide range of scales, poses, lighting, expressions, and occlusions.

[0082] The training set and the test set are mainly composed of the public dataset WiderFace dataset and the semi-supervised supplementary dataset COCO, wherein the semi-supervised dataset COCO only acts on the training set, and the test set is entirely composed of WiderFace. The WiderFace contains faces in 61 scenarios, and has corresponding label coordinate data, and the faces in COCO do not have corresponding label data as semi-supervised data.

[0083] The 158,989 labeled faces of the training set WiderFace and the semi-supervised dataset and 14320 faces in COCO constitute the MixData mixed data as the training set.

[0084] In the training set, 14320 face pictures in the COCO dataset are added, without labeled labels. Face pictures in various life scenes are included, and the data has strong robustness.

[0085] The test set WiderFace contains three levels of detection difficulty in each subset: Easy, Medium, and Hard.

[0086] Semi-supervised learning can make full use of a large number of unlabeled samples to improve learning performance under the guidance of a small number of labeled samples, avoid waste of data resources, and solve the problems of weak generalization ability of supervised learning methods when there are few labeled samples and inaccuracy of unsupervised learning methods when there is no sample label guidance. The main goal is to use the data distribution information hidden in a large number of unlabeled samples to improve the learning performance when only a small number of labeled samples are used.

[0087] Step 2: Construct a target detection network under semi-supervised learning, including: a backbone network, a feature pyramid network FPN, an SSH module, a Head module, and a prediction module;

[0088] Step 2.1: Construct the backbone network and perform three-stage feature extraction;

[0089] The backbone network contains three-stage separable convolution conv_dw modules; the convolution kernel dimension of the convolution layer in each separable convolution conv_dw module is a x a, and the step is k.

[0090] The ith face image x i and its corresponding label t i After encoding, the input feature S with a dimension of MxMxC is obtained i , i After processing by the first-stage separable convolution conv_dw module, the first-stage network feature S with a dimension of (M / p)x(M / p)x(Cxp) is obtained i_p , i_p After processing by the second-stage separable convolution conv_dw module, the second-stage network feature S with a dimension of (M / q)x(M / q)x(Cxq) is obtained i_q , _q After processing by the third-stage separable convolution conv_dw module, the third-stage network feature S with a dimension of (M / r)x(M / r)x(Cxr) is obtained i_r , wherein p represents the feature compression ratio of the first stage of the network, q represents the feature compression ratio of the second stage of the network, and r represents the feature compression ratio of the third stage of the network;

[0091] The proposed method is suitable for the current mainstream backbone network, and experimental verification is performed on various networks to analyze the effectiveness of the method.

[0092] Step 2.2: Constructing a feature pyramid FPN and performing image feature extraction;

[0093] The feature pyramid FPN adjusts the channel number of the three features S , and S , respectively, and then performs upsampling operations on the three adjusted features, respectively, to obtain a first upsampled feature S , a second upsampled feature S , and a third upsampled feature S ; Fusing S i_qr , S , and S obtains a first fused feature S i_pq ;

[0094] Step 2.3: Strengthening feature extraction through an SSH module;

[0095] The SSH module performs a splicing operation on the features S i_pq , S i_qr , and S to obtain a spliced feature S i_pqr ;

[0096] The SSH module utilizes three parallel convolutions with kernel sizes of dxd, ex e, and fxf, respectively, on the stitched feature S i_pqr is processed, and the first-stage enhanced feature S i_puut , the second-stage enhanced feature S i_qout , and the third-stage enhanced feature S i_rout are obtained accordingly.

[0097] The SSH detects faces from three different feature maps, and the three modules are used to detect small, medium, and large faces, respectively. Each detection module in the SSH contains a convolutional binary classifier and a regressor for detecting and locating faces, similar to the FPN.

[0098] In the context module, two different-sized convolutional kernels are used to model the context in this way, which increases the receptive field of the corresponding layer, is proportional to the detection of the corresponding layer, and is therefore proportional to the target scale of each detection module, while also increasing the target scale in each detection module. To reduce the number of model parameters, the method of depth separable convolution is used, and 3x3 convolutional kernels are used instead of 5x5 and 7x7.

[0099] Step 2.4: Construct the Head module and extract features for classification, predicted frames, and key points.

[0100] The Head module combines S i_pout , S i_qout , and S i_rout into a predicted feature S i_new , and then performs tensor reconstruction operations of three dimensions on S i_new , respectively, to form a face category feature S i_class with U channels, a face frame feature S i_box with V channels, and a face key point feature S i_lands with W channels.

[0101] Step 2.5: Establish a loss function.

[0102] The total loss function L i of the i-th face image x i is constructed using equation (1).

[0103] L i = w i_class x L i_class + w i_box x L i_box + w i_lands x L i_lands (1)

[0104] In equation (1), L i_classthe i-th face image x i the average absolute and mean squared error loss of face class, L i_box the i-th face image x i the average absolute and mean squared error loss of face bounding box, L i_lands the i-th face image x i the average absolute and mean squared error loss of face key points, w i_class the i-th face image x i the loss weight of face class, w i_box the i-th face image x i the loss weight of face bounding box, w i_lands the i-th face image x i the loss weight of face key points;

[0105] L i_class , L i_box and L i_Lands are respectively calculated according to the average absolute and mean squared error loss of face class feature S i_class , face bounding box feature S i_box and face key points feature S i_lands . The average absolute and mean squared error loss is a combination of the average absolute error loss and the mean squared error loss.

[0106] Step 2.6: modification and decoding of the prediction module;

[0107] The prediction module concatenates S i_class , S i_box and S i_lands to obtain the final prediction feature S i_out with a channel number of (U+V+W), and then decodes S i_out to obtain the prediction result of the i-th face image x i containing the face bounding box and the face key points;

[0108] The non-maximum suppression is used to remove the prediction boxes with high overlap in the prediction result, thereby obtaining the final prediction result f i of the i-th face image x i ; the position of any face bounding box in the final prediction result f i is denoted as (x i , y i , w i , h i ), wherein (x i , y i ) is the center point coordinate of the face bounding box, and w i , h i correspond to the width and height of the face bounding box; the prediction result f iThe face key point set of a human face in the image is {(x ih ,y ih )|h=1,2,…,H}, wherein x ie ,y ie represent the hth key point of the face, and H represents the number of key points; and finally, the final result is obtained by removing duplicate detection values through NMS (non-maximum suppression). Compared with the pseudo labels created in the image classification task, the pseudo labels created by object detection are more complex, because an image usually contains multiple detected target objects, the annotations of which consist of positions and categories. Moreover, the annotations of the face data in the Widerface dataset include categories, detection boxes, and five key position points. Given an unlabeled image, thousands of candidate boxes are predicted by detecting objects through a teacher model. Then, NMS (non-maximum suppression) is performed to eliminate redundant detection boxes. Most of the redundant boxes are removed through NMS, but some non-foreground detection boxes still exist, so only the candidate boxes with foreground scores higher than a threshold are retained as pseudo boxes.

[0109] Step 3: based on the face image dataset X, the target detection network is trained by using the gradient descent method, and the total loss function L i is calculated to update the network parameters until the total loss function L i converges, so as to obtain the trained target detection network as the teacher model G.

[0110] Step 4: constructing pseudo labels of the semi-supervised learning dataset;

[0111] The unlabeled face image dataset Y is input into the teacher model G for processing, and the pseudo labels F Y of Y are obtained; wherein F j ={fy1,fy2…fy m …fy j}, and fy j represents the predicted label corresponding to the unlabeled image data y

[0112] After mixing the labeled face image dataset X and the pseudo-labeled face image dataset Y, a new face image dataset XY={xy1,xy2…xy g …xy m+n} is obtained, wherein xy g represents the gth new face image, and the new face image dataset XY corresponds to a new label set TF={tf1,tf2…tf g …tf m+n}, wherein tf g represents the label of the gth new face image xy g , and g=1,2,3…m+n, m+n is the total number of images in the new face image dataset XY.

[0113] The classical method of adding pseudo-labels 1. First split the labeled dataset into a training set and a test set. Then, train a detection algorithm on the labeled training data. 2. Use the trained classifier to predict the class labels for all unlabeled data instances. The highest accuracy among these predicted class labels is considered the "pseudo-label". 3. Connect the "pseudo-labeled" data with the correctly labeled training data. Retrain the detector on the combined "pseudo-labeled" and correctly labeled training data. 4. Use the trained classifier to predict the class labels for the labeled test data instances. Evaluate the detector performance using the metric of your choice. 5. Repeat 1 through 4 until the predicted class labels in 2 no longer meet a particular probability threshold, or until no more unlabeled data remains.

[0114] During the training process of the algorithm, directly using all unlabeled samples not only affects the performance of the algorithm, but also significantly reduces the computational efficiency of the algorithm. The commonly used semi-supervised learning algorithm has high time complexity, can only process small-scale data, and has poor scalability. Therefore, under the premise of effectively utilizing unlabeled samples, how to efficiently use a large number of unlabeled samples to improve the performance and scalability of the algorithm is a problem that needs to be solved when using semi-supervised learning technology to process large-scale data.

[0115] Step 5: Construct a new semi-supervised detection network and train it, which is composed of the backbone network in the teacher model G, the SSH module, the prediction module, and the new pyramid module RE_BiFPN;

[0116] Step 5.1: Three-stage feature extraction based on the backbone network;

[0117] The gth new face image xy g and its corresponding label tf g After being input into the backbone network of the teacher model G for processing, the new first-stage network feature S g_p , the new second-stage network feature S g_p , and the new third-stage network feature S g_r are obtained;

[0118] Step 5.2: Construct a new pyramid module RE_BiFPN and perform image feature extraction;

[0119] Step 5.2.1: The new pyramid module RE_BiFPN adjusts the channel numbers of S g_p , S g_p , and S g_r using a convolution with a dimension of bxb, to obtain the adjusted features and And respectively after two two fusion, get three fusion characteristics As Figure 2 shown, the dashed line in the figure represents the iteration step;

[0120] The new feature pyramid module RE_BiFPN uses the convolution with the dimension of bxb to adjust the channel number of S g_p , S g_p and S g_r , and get the features and Three features are fused by the following method. and In Figure 2 represent the features C (k-2) , C (k-1) and C k .

[0121] The feature is convolved by separable deep convolution conv to get the first layer feature of the third stage Figure 2 In C ktd1 represent the feature The features and are fused by the double layer feature fusion formula, and the first layer feature of the second stage is obtained Figure 2 In C (k-1)td1 represent the feature

[0122] The features and are fused by the double layer feature fusion formula, and the first layer feature of the first stage is obtained Figure 2 In C (k-2)td1 represent the feature The features and are fused by the double layer feature fusion formula, and the second layer feature of the first stage is obtained Figure 2 In C (k-2)td2 represent the feature The features and are fused by the double layer feature fusion formula, and the third layer feature of the first stage is obtained

[0123] The features and are fused by the double layer feature fusion formula, and the first layer feature of the second stage is obtained Figure 2Mid-C (k-1)td1 representative feature fuse the features by using a three-layer feature fusion formula features and features fuse, and obtain the second layer feature of the second stage Figure 2 Mid-C (k-1)td2 representative feature fuse the features by using a two-layer feature fusion formula and features fuse, and obtain the third layer feature of the second stage

[0124] fuse the features by using a three-layer feature fusion formula features and features fuse, and obtain the second layer feature of the third stage Figure 2 Mid-C ktd2 representative feature fuse the features by using a two-layer feature fusion formula and features fuse, and obtain the third layer feature of the third stage

[0125] take the features as the input of the second layer of feature fusion, and obtain the final three-stage fused features after iterative fusion Figure 2 Mid-C (k-2)out ,C (k-1)out and C kout respectively represent the features

[0126] Step 5.2.2, fuse the three fused features again, and obtain the three-stage fused features after two-by-two fusion

[0127] In order to solve the problem of multi-scale change in object detection, many networks use single high-level features, but this has a obvious defect, that is, small objects have less pixel information, which is easily lost in the down-sampling process. In order to solve the problem of obvious size difference of objects in detection, the image pyramid method is used to enhance multi-scale change, which has good effect, but brings great computational complexity. Therefore, the FPN feature pyramid network structure can solve the problem of multi-scale change in object detection with little increase in computational complexity.

[0128] ​Different feature layers correspond to anchors of different scales. Each feature layer has different scale information relative to the original image, so the scale information in the original features is separated, and each feature layer only processes single scale information.

[0129] As Figure 2 shown, three different scale features are selected for cross fusion, and then iterated twice to obtain the final desired result. The new RE_BiFPN structure realizes repeated bidirectional cross-scale connection of features, as well as weighted feature fusion mechanism and feature iteration cycle mechanism.

[0130] Step 5.3: Enhanced feature extraction through the SSH module;

[0131] The SSH module in the teacher model G fuses the three stage fusion features and performs a splicing operation to obtain spliced features S g_pqr ;

[0132] The SSH module processes the spliced features S g_pqr using three parallel convolutions with kernel sizes of dxd, exd, and fxd, respectively, to obtain new first-stage enhanced features S g_pout , new second-stage enhanced features S g_qout , and new third-stage enhanced features S g_rout ;

[0133] Step 5.4: Classification, prediction box, and key point feature extraction based on the Head module;

[0134] The Head module combines S g_pout , S g_qout , and S g_rout into new prediction features S g_new , and then performs tensor reconstruction operations of three dimensions on S g_new to obtain new face class features S g_class , new face box features S g_box , and new face key point features S g_lands ;

[0135] Step 5.5: Establish the total loss function L new of the new semi-supervised detection network using formula (2):

[0136] L new = αL i + βL g (2)

[0137] In formula (2), L gThe loss function of the new semi-supervised detection network for the gth new face image xy g , alpha represents the weight of the labeled image supervision loss, and beta represents the weight of the pseudo-labeled image supervision loss.

[0138] Step 5.6: Correction and decoding of the prediction module;

[0139] The prediction module in the teacher model G will S g_class , S g_box and S g_lands are spliced to obtain a new final prediction feature S g_out , and after decoding S g_out , the gth new face image xy g is obtained, which contains the prediction results of the face frame and the face key points.

[0140] Step 6: Based on the face image dataset XY, the gradient descent method is used to train the new semi-supervised detection network, and the total loss function L new is calculated to update the network parameters until the total loss function L new converges, thereby obtaining the trained target detection network and serving as a new detection model G new .

[0141] After the new semi-supervised training, the final face detection model is obtained, and the F new ={f1, f2…f i …f n+m} label effect accuracy is more in line with the real label.

[0142] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above small target detection method, and the processor is configured to execute the program stored in the memory.

[0143] In this embodiment, a computer readable storage medium stores a computer program on the computer readable storage medium, and the computer program is executed by a processor to execute the steps of the above small target detection method.

[0144] The present application performs some ablation experiments on different backbone networks as shown in Table 1, and the labeled dataset uses the Widerface dataset, and the unlabeled dataset uses the face data in COCO, and the AP is used as the evaluation index.

[0145] Table 1: Semi-supervised accuracy results under different feature fusion layers

[0146]

[0147] From table 1, it can be seen that the method of the present application has better results in detection accuracy under each network.

Claims

1. A small target detection method based on semi-supervised and feature fusion, characterized by: Follow these steps: Step 1: Construct a face image dataset and perform preprocessing; Step 1.1: Obtain a labeled dataset of real face images, perform size normalization and preprocessing to obtain a preprocessed face image dataset of dimension M×M×C, denoted as X={x1,x2,...,x i , ..., x n }, where x i Let the i-th face image be represented by the label set T = {t1, t2, ..., t3}. i , ...t n }, where t i Let x represent the i-th face image. i The corresponding labels are i = 1, 2, ..., n, where n is the total number of images in the labeled face image dataset; M represents the length and width of the processed image, and C represents the number of channels; Step 1.2: Obtain the unlabeled face image dataset, perform size normalization and preprocessing, and obtain the preprocessed face image dataset with dimensions M×M×C, denoted as Y={y1,y2,...,y j , ...y m }, where y j Let j represent the j-th unlabeled face image, where j = 1, 2, ..., m, and m is the total number of images in the unlabeled face image dataset. Step 2: Construct a semi-supervised learning-based object detection network, including: backbone network, feature pyramid (FPN), SSH module, head module, and prediction module; Step 2.1: Construct the backbone network and perform three-stage feature extraction; The backbone network contains three-stage separable convolutional conv_dw modules; the kernel size of the convolutional layer in each separable convolutional conv_dw module is a×a, and the stride is k; The labeled i-th face image x i and its corresponding tag t i After encoding, the input feature S with dimensions M×M×C is obtained. i S i After processing by the first-stage separable convolution conv_dw module, the first-stage network features S are obtained with dimensions of (M / p)×(M / p)×(C×p). ip S ip After further processing by the second-stage separable convolution conv_dw module, the second-stage network features S with dimensions of (M / q)×(M / q)×(C×q) are obtained. i_q Finally S i_q After processing by the third-stage separable convolution conv_dw module, the network feature S of the third stage with a dimension of (M / r)×(M / r)×(C×r) is obtained. i_r Where p represents the feature compression ratio of the first stage, q represents the feature compression ratio of the second stage, and r represents the feature compression ratio of the third stage; Step 2.2: Construct the Feature Pyramid (FPN) and extract image features; The Feature Pyramid Network (FPN) utilizes b×b ​​convolution pairs. and After adjusting the number of channels, upsampling is then performed on the three adjusted features to obtain the first upsampled feature. Second upsampling feature and third upsampling features Will and The first fusion feature S is obtained after fusion. i_qr ,Will and The second fusion feature S is obtained after fusion. i_pq ; Step 2.3: Enhance feature extraction using the SSH module; The SSH module will use feature S i_pq S i_qr and After performing the splicing operation, the splicing feature S is obtained. i_pqr ; The SSH module utilizes three parallel convolutions with kernel sizes of d×d, e×e, and f×f to concatenate the feature S. i_pqr After processing, the first-stage enhanced feature S is obtained. i_pout The second stage strengthens the feature S i_qout and the third stage enhanced feature S i_rout ; Step 2.4: Construct the Head module and extract features from the classification, prediction boxes, and keypoints; The Head module will S i_pout S i_qout and S i_rout Combined into predictive features S i_new Then S i_new Tensor reconstruction operations are performed in three dimensions to form a face category feature S with U channels. i_class Face bounding box feature S with V channels i_box Facial landmark features S with W channels i_lands ; Step 2.5: Establish the loss function; Using Equation (1), an object detection network is constructed for the i-th face image x. i Total loss function L i ; L i =w i_class ×L i_class +w i_box ×L i_box +w i_lands ×L i_lands (1) In equation (1), L i_class Let x represent the i-th face image. i Mean absolute and mean squared error loss for face categories, L i_box Let x represent the i-th face image. i Mean absolute and mean squared error loss of face bounding boxes, L i_lands Let x represent the i-th face image. i Mean absolute and mean squared error loss of facial landmarks, w i_class w represents the face category loss weights for the i-th face image xi. i_box Let x represent the i-th face image. i Face bounding box loss weights, w i_lands Let x represent the i-th face image. i Loss weights for facial landmarks; Step 2.6: Correction and decoding of the prediction module; The prediction module will S i_class S i_box and S i_lands After concatenation, the final predicted feature S with (U+V+W) channels is obtained. i_out Then for S i_out After decoding, the i-th face image x is obtained. i The prediction results include face bounding boxes and facial landmarks; Non-maximum suppression is used to remove prediction boxes with high overlap in the prediction results, thus obtaining the i-th face image x. i The final prediction result f i Let the final prediction result f i The position of any face bounding box in the image is denoted as (x). i y i w i h i ), where (x i y i ) represents the coordinates of the center point of the face bounding box, w i h i The width and height of the corresponding face bounding box; let the prediction result f i The set of facial key points in the image is {(x ih y ih )|h=1,2,...,H}, where x ie y ie These represent the h-th key point of the face, where H represents the number of key points; Step 3: Based on the face image dataset X, train the object detection network using gradient descent and calculate the total loss function L. i To update the network parameters until the total loss function L is reached. i The training continues until convergence, thus obtaining the trained object detection network and using it as the teacher model G. Step 4: Construct pseudo-labels for the semi-supervised learning dataset; The unlabeled face image dataset Y is input into the teacher model G for processing to obtain the pseudo-label F of Y. Y ={fy1, fy2...fy j ...fy m }; where fy j Represents unlabeled image data y j Corresponding predicted labels; The labeled face image dataset X and the pseudo-labeled face image dataset Y are combined to form a new face image dataset XY = {xy1, xy2, ..., xy}. g ...xy m+n }, where xy g Let the g-th new face image be denoted as tf1, tf2, ..., tf2. Let the new face image dataset XY correspond to the new label set TF = {tf1, tf2, ..., tf3}. g ...tf m+n }, tf g Let xy represent the g-th new face image. g The labels are: g = 1, 2, 3...m+n, where m+n is the total number of images in the new face data image set XY; Step 5: Construct and train a novel semi-supervised detection network, which consists of the backbone network, SSH module, prediction module, and new pyramid module RE_BiFPN in the teacher model G; Step 5.1: Three-stage feature extraction based on the backbone network; The g-th new face image xy g and its corresponding tag tf g After being processed in the backbone network of the teacher model G, new first-stage network features S are obtained. g_p New second-stage network features S g_p The new third-stage network feature S g_r ; Step 5.2: Construct a new pyramid module RE_BiFPN and perform image feature extraction; Step 5.2.1: The new pyramid module RE_BiFPN utilizes convolution pairs S of dimension b×b. g_p S g_p and S g_r After adjusting the number of channels respectively, the adjusted features are obtained. and After pairwise fusion, three fusion features were obtained. Step 5.2.2: Combine the three features After merging each pair separately, a fusion is obtained, resulting in three stages of fusion characteristics. Step 5.3: Enhanced feature extraction using the SSH module; The SSH module in the teacher model G integrates the features of the three stages. and After performing the splicing operation, the splicing feature S is obtained. g_pqr ; The SSH module uses three parallel convolutional pairs with kernel sizes of d×d, e×e, and f×f to concatenate the feature S. g_pqr After processing, the new first-stage enhanced feature S is obtained. g_pout New second phase enhanced features S g_qout And the new third stage enhanced feature S g_rout ; Step 5.4: Extract features from the classification, prediction boxes, and keypoints based on the Head module; The Head module will S g_pout S g_qout and S g_rout Combined into new predictive features S g_new Then S g_new Perform tensor reconstruction operations in three dimensions to obtain new face category features S. g_class New face bounding box feature S g_box and new facial landmark features S g_lands ; Step 5.5: Use equation (2) to establish the total loss function L of the novel semi-supervised detection network. new : L new =αL i +βL g (2) In equation (2), L g This indicates that the novel semi-supervised detection network performs a test on the g-th new face image xy. g The loss function is denoted by α, which represents the weight of the labeled image supervision loss, and β, which represents the weight of the pseudo-labeled image supervision loss. Step 5.6: Correction and decoding of the prediction module; The prediction module in the teacher model G will S g_class S g_box and S g_lands The new final predicted feature S is obtained by splicing. g_out Then for S g_out After decoding, the g-th new face image xy is obtained g The prediction results include face bounding boxes and facial landmarks; Step 6: Based on the face image dataset XY, train the novel semi-supervised detection network using gradient descent and calculate the total loss function L. new To update the network parameters until the total loss function L is reached. new The training continues until convergence, resulting in the trained object detection network, which serves as the novel detection model G. new .

2. The small target detection method based on semi-supervised and feature fusion according to claim 1, characterized in that, The pairwise fusion process in step 5.2.1 includes: Features After performing separable depthwise convolution (conv), the first layer features of the third stage are obtained. Using a two-layer feature fusion formula for features and characteristics The features are then fused to obtain the first layer of features in the second stage. Using a two-layer feature fusion formula for features and characteristics The features are fused together to obtain the first layer of features in the first stage. Using a two-layer feature fusion formula for features and characteristics The features are then fused to obtain the second layer of features from the first stage. Using a two-layer feature fusion formula for features and characteristics The components are fused to obtain the third layer of features from the first stage. Using a two-layer feature fusion formula for features and characteristics The features are then fused to obtain the first layer of features in the second stage. Using a three-layer feature fusion formula to analyze features feature and characteristics The features are then fused to obtain the second layer features of the second stage. Using a two-layer feature fusion formula for features and characteristics The components are fused to obtain the third layer of features in the second stage. Using a three-layer feature fusion formula to analyze features feature and characteristics The fusion process is performed to obtain the second layer of features in the third stage. Using a two-layer feature fusion formula for features The features are fused together to obtain the third layer of features in the third stage.

3. The small target detection method based on semi-supervised and feature fusion according to claim 2, characterized in that, The pairwise fusion process in step 5.2.2 is to... As an adjusted feature and After processing according to the fusion method in step 5.2.1, three-stage fusion features are obtained.

4. The small target detection method based on semi-supervised and feature fusion according to claim 2 or 3, characterized in that, The formula for the two-layer feature fusion is as follows: In equation (3), S one S two There are two input features, w one w two It is S one S two The corresponding learning parameters are: Resize represents the downsampling operation; ε is the bias hyperparameter.

5. The small target detection method based on semi-supervised and feature fusion according to claim 2 or 3, characterized in that, The three-layer feature fusion formula is as follows: In equation (4), S one S two S three There are three input features, w one w two w three It is S one S two S three The corresponding learning parameters are: Resize represents the downsampling operation; ε is the bias hyperparameter.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the small target detection methods of claims 1-5, and the processor is configured to execute the programs stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the small target detection methods described in claims 1-5.

Images