A semi-supervised object detection method with improved pseudo-label quality and class imbalance

Abstract

The present invention discloses a semi-supervised target detection method for improving pseudo-label quality and category imbalance, comprising: constructing a semi-supervised target detection model, including a teacher network and a student network based on a target detection network; discretizing the regression branch of the prediction offset in the target detection network; constructing a training loss function of the target detection network, thereby obtaining an overall loss function of the student network; obtaining a fixed position confidence based on the variance of the position offset confidence, combining the fixed position confidence with the classification confidence, and constructing a comprehensive confidence; using the comprehensive confidence and weighting the overall loss function to alleviate the adverse effects of category imbalance on target detection; the student network is updated by reverse stochastic gradient descent according to the loss function, and the teacher network is updated by using the EMA of the student network. On the basis of proposing the comprehensive confidence, the present invention introduces a class-related comprehensive confidence to weight the loss function, thereby alleviating the influence of category imbalance on the performance of the semi-supervised target detection model.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, semi-supervision and target detection, and in particular to a method for alleviating category imbalance and improving pseudo-label quality by combining localization confidence in a semi-supervised target detection algorithm. Background Art

[0002] Object detection is an important task in computer vision. Currently, most object detection tasks are based on deep learning algorithms, and the performance depends on the amount of labeled data in model training, such as the location and category of objects. However, the high cost of manual labeling requires relevant researchers to balance model performance and development costs when conducting experimental research. Therefore, semi-supervised learning algorithms have emerged to use unlabeled data to further improve the performance of model object detection.

[0003] How to make full use of a large number of unlabeled data types (such as images, texts, etc.) is the main task that semi-supervised object detection needs to solve. The current mainstream semi-supervised object detection technology can be divided into consistency learning-based and pseudo-label learning algorithms. Among them, the semi-supervised object detection algorithm based on consistency learning uses a dual deep convolutional neural network to identify the consistency features between different perturbations of the same unlabeled information image (such as horizontal flipping, different contrast, brightness, etc.), so as to utilize the information of unlabeled image data. The semi-supervised object detection algorithm based on pseudo-label learning uses a pre-trained model learned on labeled data (called a teacher network) to infer the unlabeled data. After NMS (non-maximum suppression) reduces a large number of redundant frames, a threshold is used to select the pseudo-label, and finally the pseudo-label and labeled data are used to re-train the model together (called a student network at this time). The loss function of the semi-supervised object detection task consists of the loss of labeled data and the loss of unlabeled data.

[0004] Most of the current semi-supervised target detection algorithms follow the semi-supervised image classification algorithms, using improved loss functions, data enhancement, and model training. The most advanced semi-supervised target detection algorithms combine pseudo-label learning and consistency learning techniques. However, in the pseudo-label learning module, these algorithms generally only use the classification confidence obtained by the teacher network to predict the unlabeled data as a criterion for measuring the quality of pseudo-labels. Studies have shown that there is no obvious positive correlation between classification confidence and target positioning accuracy. It often shows high classification confidence but low IOU (IOU: the intersection of predicted box and true box). Therefore, considering only classification confidence cannot obtain high-quality pseudo-labels in semi-supervised target detection, thus affecting model performance. On the other hand, semi-supervised learning faces a large amount of unlabeled data. The natural class imbalance in these data will definitely affect model learning and produce class bias. The model has a good learning ability for common classes, and predicts low confidence for uncommon classes, which will be filtered out during pseudo-label threshold screening. This is also not considered by most current algorithms. Summary of the invention

[0005] The purpose of the present invention is to provide a semi-supervised target detection method for improving pseudo-label quality and class imbalance, by improving the quality of pseudo-labels introduced by improving the network output form, and by adopting a class correlation coefficient weighted loss function to alleviate the impact of data class imbalance.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] A semi-supervised object detection method for improving pseudo-label quality and class imbalance, comprising:

[0008] Constructing a semi-supervised target detection model, the model comprising a teacher network and a student network based on a target detection network;

[0009] The regression branch of the predicted offset in the target detection network is discretized, and the confidence of each offset segment is calculated, thus realizing the confidence of each segment of the predicted offset in the grid divided by the sample image; the loss function used to train the positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct the training loss function of the target detection network, thereby obtaining the overall loss function of the student network;

[0010] Combined with variance analysis, the location confidence is obtained based on the variance of the location offset confidence, and the normalized location confidence is combined with the classification confidence to construct a comprehensive confidence. The overall loss function of the class-weighted student network is introduced by using the comprehensive confidence to alleviate the adverse effects of class imbalance on target detection.

[0011] The student network is updated by reverse stochastic gradient descent according to the loss function, and the teacher network is updated by the EMA of the student network;

[0012] When applied, for a picture to be detected, it is input into the trained student network, and the target detection result is obtained through the output of the student network.

[0013] Furthermore, during training, the teacher network:

[0014] First, a batch of data is formed by sampling according to a preset ratio, and the unlabeled image samples in the batch of data are enhanced using weak data enhancement technology;

[0015] Secondly, the unlabeled image samples are input into the randomly initialized target detection network, namely the teacher network, to obtain the prediction results of the unlabeled image samples. The prediction results are the classification confidence and position offset confidence of the target in the unlabeled image samples. The comprehensive confidence is obtained by combining the classification confidence and the positioning confidence.

[0016] Thirdly, NMS processing is performed according to the comprehensive confidence, and a threshold τ is set to filter the bounding boxes after NMS processing, that is, the prediction results of the bounding boxes after NMS processing whose comprehensive confidence is greater than the threshold are retained.

[0017] Furthermore, during training, the student network:

[0018] Weak data enhancement is performed on labeled image samples in batch data, and strong data enhancement is performed on unlabeled image samples with pseudo labels; the strong enhancement is a Mixup operation, which mixes unlabeled image samples with pseudo labels with labeled image samples in proportion to form an enhanced data set; then, the enhanced data set is input into a target detection network with the same random initialization parameters as the aforementioned teacher network, namely, a student network; the student network is then updated by performing stochastic gradient descent on the overall loss function weighted by the comprehensive confidence, and the teacher network is updated by the exponential moving average EMA of the student network; when the student network converges, the training is completed and it is used alone for target detection in images.

[0019] Furthermore, the step of discretizing the regression branch of the predicted offset in the target detection network and calculating the confidence of each segment of the offset realizes the confidence of each segment of the predicted offset in the grid divided by the sample image, including:

[0020] The object detection network adopts Faster R-CNN; based on Faster R-CNN, the prediction offset regression head part is discretized and improved, and the offset regression head is converted into a classification head; that is, the labeled image samples / unlabeled image samples are divided into multiple grids, each grid is divided into multiple segments, and the median of each segment is taken as the approximate value of the discrete offset in the segment, and the confidence of the offset of each segment is reflected as the probability that the offset is located in the segment.

[0021] Furthermore, the image samples in the enhanced dataset are input into the student network. After being processed and output by the RPN region extraction network in the student network, the upper left coordinates (x, y) and lower right coordinates (a, b) of the real bounding box and the pseudo-label bounding box are calculated online according to formulas (1) and (2), corresponding to the offset of the filtered bounding box output by the RPN region extraction network:

[0022]

[0023]

[0024] Among them G x , G y are the horizontal and vertical coordinates of the true bounding box or the horizontal and vertical coordinates of the pseudo-label bounding box, respectively. x , P y are the horizontal and vertical coordinates of the bounding box after filtering output by the corresponding RPN network, P w , P h The width and height of the filtered bounding box output by the RPN network;

[0025] The offset calculation result is downsampled and returned to the coordinate system of the last layer output feature map of Faster R-CNN. This downsampled result is compared with the grid to which the filtered bounding box output by the RPN network belongs. Information 1 is recorded at the corresponding position and information 0 is recorded in other places, forming a one-hot encoding of the position offset information, which is used for subsequent training to calculate the positioning loss.

[0026] Furthermore, the loss function used for training positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct a training loss function for the target detection network, thereby obtaining the overall loss function of the student network, including:

[0027] The overall loss function calculation of the student network is represented by the standard semi-supervised loss function, and the overall loss function is defined as shown in formula (3):

[0028] loss = l s +λl u (3)

[0029] Among them l s Represents the loss part of the labeled image sample, l u represents the loss part of the unlabeled image sample with pseudo labels, λ represents the weighting coefficient, and takes values ​​in the closed interval [1,2];

[0030] l s and l u The classification and positioning loss calculation formulas have the same form, as shown in Equation 4 and Equation 5:

[0031]

[0032]

[0033] Where L cls represents the classification loss, L loc represents the positioning loss; x, y represent the horizontal and vertical coordinates of the upper left corner of the bounding box, a, b represent the horizontal and vertical coordinates of the lower right corner of the bounding box; loc represents the subscripts of each segment divided by each grid, cls represents the category subscript; pos represents the subscript of the bounding box in the filtered bounding box output by the RPN network whose IOU with the real bounding box and the pseudo-label bounding box is greater than 0.7, N pos Indicates the number of bounding boxes with subscript pos; B ij With B ijk The value of is {0,1}, where 1 means existence and 0 means non-existence;

[0034] When calculating the positioning loss, B ijk = 1 indicates whether the offset of the j-coordinate of the i-th bounding box is located in the k-th segment in the discretized grid; when calculating the classification loss, B ij =1 means that there is a target of the jth type in the i-th bounding box; P ij represents the category score of the jth category of the i-th bounding box output by the student network, and this is used as the classification confidence; P ijk represents the confidence of the offset of the j-th coordinate in the i-th bounding box in the k-th segment, and s(·) represents the sigmoid function;

[0035] For the classification loss function L cls And the positioning loss function L loc By performing linear combination, we can get the training loss function of labeled image samples and unlabeled image samples, that is:

[0036] l s or u =L cls +ηL loc (6)

[0037] Where η represents the weighting coefficient, which ranges from 0 to 1.

[0038] Furthermore, the combined variance analysis obtains the location confidence based on the variance of the offset, combines the normalized location confidence with the classification confidence, and constructs a comprehensive confidence, including:

[0039] First, according to the output value of the student network, the classification confidence and position offset confidence are obtained; the calculation formula of the positioning confidence is as follows:

[0040]

[0041]

[0042] where L′ conf represents the fixed-position reliability, D(i) represents the variance of variable i, i j represents the confidence that the i position offset is located in the jth segment in the grid, represents the confidence mean of the i-coordinate offset; n represents the number of segments of the grid;

[0043] Using the normalized location reliability L conf , defined by formula (9):

[0044]

[0045] Where s represents the sigmoid function;

[0046] Next, the classification confidence and location confidence are combined to obtain the comprehensive confidence B score ,Right now:

[0047] B score =αcls score +(1-α)L conf (10)

[0048] where cls score represents the classification confidence, which is obtained by the prediction output of the classification branch of the teacher network, and α represents the weighting coefficient, which ranges from 0 to 1;

[0049] The comprehensive confidence calculated by formula (10) is used as a new standard for NMS processing and threshold filtering to obtain pseudo labels for unlabeled image samples containing positioning and classification accuracy information.

[0050] Furthermore, the overall loss function of the class-weighted student network using comprehensive confidence is introduced to alleviate the adverse effects of class imbalance on target detection, including:

[0051]

[0052]

[0053] where c i represents a certain category subscript, and c represents all category subscripts; Represents the sum of the predicted comprehensive confidence of a certain category of unlabeled image samples, Represents the sum of the predicted comprehensive confidence of the unlabeled image samples; the comprehensive confidence related to the category Weighted with the overall loss function, formula (12) gives the weighted result of the overall loss function.

[0054] Compared with the prior art, the present invention has the following technical features:

[0055] The present invention designs a comprehensive confidence for target detection tasks, which reflects the quality of predicted bounding box positioning and classification, can obtain higher-quality pseudo labels, and improve the performance of the semi-supervised target detection model. At the same time, in semi-supervised learning, there is a large amount of unlabeled data that belongs to long-tail type data and has class imbalance characteristics. On the basis of proposing a comprehensive confidence, a class-related comprehensive confidence is introduced to weight the loss function, which alleviates the impact of class imbalance on the performance of the semi-supervised target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the overall architecture of the semi-supervised target detection model of the present invention;

[0057] Figure 2 Schematic diagram of the target detection network, where cxi, cyi, cai, cbi are the confidences of the offsets of the corresponding quaternion coordinates (x, y, a, b) in the i-th segment of the grid;

[0058] Figure 3 The figure is a schematic diagram of the discretization of the offset in the image grid. The right side of the figure is a magnified grid on the left side. Its height and width are divided into 5 segments, and the median value of each segment is taken as the discretized offset in a grid.

[0059] Figure 4 This is a training flow chart of the network model in the present invention. DETAILED DESCRIPTION

[0060] The present invention provides a semi-supervised target detection method for improving pseudo-label quality and class imbalance, the method comprising:

[0061] Constructing a semi-supervised target detection model, the model comprising a teacher network and a student network based on a target detection network;

[0062] The regression branch of the predicted offset in the target detection network is discretized, and the confidence of each offset segment is calculated, thus realizing the confidence of each segment of the predicted offset in the grid divided by the sample image; the loss function used to train the positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct the training loss function of the target detection network, thereby obtaining the overall loss function of the student network;

[0063] Combined with variance analysis, the location confidence is obtained based on the variance of the offset, and the normalized location confidence is combined with the classification confidence to construct a comprehensive confidence. The overall loss function of the class-weighted student network is introduced by using the comprehensive confidence to alleviate the adverse effects of class imbalance on target detection.

[0064] The student network is updated by reverse stochastic gradient descent according to the loss function, and the teacher network is updated by the EMA of the student network;

[0065] When applied, for a picture to be detected, it is input into the trained student network, and the target detection result is obtained through the output of the student network.

[0066] The present invention is applicable to all current target detection networks. The selected target detection network can be a single-stage network such as the YOLO series, the SSD series, or a two-stage network such as the Faster R-CNN series. The following embodiments will be described in detail using the two-stage target detection network Faster R-CNN as an example.

[0067] Part 1: Overall network architecture and training process

[0068] The semi-supervised target detection model of the present invention is as follows Figure 1 As shown, it includes a teacher network and a student network; the dataset used to train the semi-supervised target detection model contains both labeled image samples and unlabeled image samples.

[0069] (1) Training of the teacher network

[0070] First, the present invention samples a batch of data according to a preset ratio (for example, the proportion of unlabeled image samples is 1 / 3), and uses weak data enhancement technology on the unlabeled image samples in the batch data, including flipping, translation, etc. to perform data enhancement.

[0071] Secondly, the unlabeled image samples are input into the randomly initialized target detection network, namely the teacher network, to obtain the prediction results of the unlabeled image samples. The prediction results are the classification confidence and position offset confidence of the targets in the unlabeled image samples. The comprehensive confidence is obtained by combining the classification confidence and the positioning confidence.

[0072] Thirdly, NMS (non-maximum suppression) processing is performed according to the comprehensive confidence, and a threshold τ (τ is between 0 and 1) is set to screen the bounding boxes after NMS processing, that is, the prediction results of the bounding boxes after NMS processing whose comprehensive confidence is greater than the threshold are retained.

[0073] Finally, the prediction result based on threshold screening is the pseudo label of the unlabeled image sample.

[0074] (2) Training of student networks

[0075] Weak data enhancement is performed on labeled image samples in batch data, and strong enhancement is performed on unlabeled image samples with pseudo labels; the strong enhancement is a Mixup operation, that is, unlabeled image samples with pseudo labels are mixed with labeled image samples in proportion to form an enhanced data set; then, the enhanced data set is input into a target detection network with the same random initialization parameters as the aforementioned teacher network, namely the student network; then, the student network is updated by stochastic gradient descent through a loss function containing both labeled and unlabeled data types (using the overall loss function weighted by comprehensive confidence, formula 12), and the teacher network is updated by the exponential moving average EMA of the student network; the entire network architecture is an end-to-end iterative training architecture. When the student network converges, the training is completed and it is used alone for target detection tasks.

[0076] In practical applications, for a picture to be detected, it is input into the trained student network, and the target detection result, namely the classification confidence (classification result) and the position offset confidence (positioning result), is obtained through the output of the student network.

[0077] In the second part, the regression branch of the predicted offset in the target detection network is discretized, and the confidence of each segment of the offset is calculated, thereby realizing the confidence of each segment of the predicted offset in the grid divided by the sample image; the loss function used to train the positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct the training loss function of the target detection network, thereby obtaining the overall loss function of the student network.

[0078] The target detection network used in one embodiment of the present invention is as follows: Figure 2 As shown in Figure 2, the overall network discretizes the prediction offset regression head based on the two-stage target detection network Faster R-CNN (see Figure 2The dotted line part) converts the offset regression head into a classification head, that is, the offset regression parameter value originally outputted is converted into the confidence of the offset of each segment in the image grid; that is, the labeled image samples / unlabeled image samples are divided into multiple grids, each grid is divided into multiple segments, and the median of each segment is taken as the approximate value of the discrete offset in the segment. The confidence of the offset of each segment is reflected as the probability that the offset is in the segment. For example, taking the prediction stage as an example, one of the outputs of the target detection network is the confidence of the position offset. The segment with the highest confidence of the position offset is selected, indicating that the probability that the predicted offset falls on this segment is the highest, and the median of the segment is taken as the approximate value of the offset falling on this segment.

[0079] from Figure 3 As can be seen in , assuming that the number of segments of each grid is n = 5 (the larger n is, the more accurate the positioning is), there are 5 discrete offsets available for selection and learning, and the offset value is the median value of each segment ( Figure 3 The image samples in the enhanced dataset are input into the student network. After being processed and output by the RPN region extraction network in the student network, the coordinate position offset of the pseudo-label and the real label needs to be further calculated for the subsequent training process. According to formulas (1) and (2), the upper left coordinates (x, y) and lower right coordinates (a, b) of the real bounding box (the target bounding box of the labeled image sample, which is the inherent annotation information of the labeled image sample) and the pseudo-label bounding box (the target bounding box obtained by the teacher network for the unlabeled image sample) are calculated online, corresponding to the offset of the filtered bounding box output by the RPN region extraction network (only the coordinate position offset of the bounding box whose IOU (Intersection over Union) with the real bounding box and the pseudo-label bounding box is greater than 0.7 is calculated):

[0080]

[0081]

[0082] Among them G x , G y are the horizontal and vertical coordinates of the true bounding box or the horizontal and vertical coordinates of the pseudo-label bounding box, respectively. x , P y are the horizontal and vertical coordinates of the bounding box after filtering output by the corresponding RPN network, P w , P hThe width and height of the bounding box after filtering output by the RPN network. The offset calculation result is downsampled and returned to the coordinate system of the output feature map of the last layer of Faster R-CNN. This downsampled result is compared with the grid (after discretization) to which the bounding box after filtering output by the RPN network belongs. Information 1 is recorded at the corresponding position, and information 0 is recorded in other places, forming a one-hot encoding of the position offset information, which is used for subsequent training to calculate the positioning loss (see Formula 4).

[0083] For the downsampling result compared with the grid, Figure 3 For example, its 2*2 grid is the last layer output feature map after downsampling. After scaling the offset with the same downsampling ratio, find a grid to which the bounding box to calculate the offset belongs. This grid has been discretized (i.e., the length and width are each divided into 5 segments). Observe which segment the offset value after scaling falls on, and take the median of the segment as the approximate value of the offset after scaling. And the corresponding position on the coordinate position offset confidence label is set to 1, indicating that the offset falls on this segment. In fact, the discretized grid set by the present invention allows the bounding box coordinates belonging to the grid to be further moved and adjusted by up to n pixels from a numerical analysis, where n is the downsampling ratio. For example: 01000, which means that the offset value is located in the second segment, and the offset value is the median of the second segment.

[0084] The overall loss function calculation of the student network is represented by the standard semi-supervised loss function, and the overall loss function is defined as shown in formula (3):

[0085] loss = l s +λl u (3)

[0086] Among them l s Represents the loss part of the labeled image sample, l u It represents the loss part of the unlabeled image sample with pseudo label, λ represents the weighting coefficient, and takes value in the closed interval [1,2].

[0087] In the present invention, taking Faster R-CNN as an example of the target detection network, since the regression head of the original model is changed into a classification head, the confidence of the predicted offset in each segment in the grid is achieved.

[0088] Therefore, the loss function used to train the positioning accuracy is transformed from the smooth L1 bounding box regression loss function to the binary cross entropy classification loss function (see formula (4)). At this time, the functions used to calculate the positioning loss (see formula (4)) and the classification loss (see formula (5)) are unified into the binary cross entropy loss function. In the semi-supervised object detection algorithm, l s and l uThe classification and positioning loss calculation formulas have the same form, as shown in Equation 4 and Equation 5:

[0089]

[0090]

[0091] Where L cls represents the classification loss, L loc represents the positioning loss; x, y represent the horizontal and vertical coordinates of the upper left corner of the bounding box, a, b represent the horizontal and vertical coordinates of the lower right corner of the bounding box. loc represents the subscript of each segment of each grid division, cls represents the category subscript. pos represents the subscript of the bounding box in the filtered bounding box output by the RPN network whose IOU with the real bounding box and the pseudo-label bounding box is greater than 0.7, N pos Indicates the number of bounding boxes with subscript pos. ij With B ijk The value of is {0,1}, where 1 means existence and 0 means non-existence. When calculating the positioning loss, B ijk Indicates whether the j-coordinate offset in the predicted target bounding box i belongs to the k-th segment, that is, B ijk = 1 indicates whether the offset of the j-coordinate of the i-th bounding box is located in the k-th segment in the discretized grid; when calculating the classification loss, B ij Indicates whether there is a j-th target in the predicted target bounding box i, that is, B ij =1 means that there is a target of the jth type in the i-th bounding box. ij represents the category score of the jth category of the i-th bounding box output by the student network, and this is used as the classification confidence; P ijk represents the confidence of the offset of the j-th coordinate in the i-th bounding box in the k-th segment. s(·) represents the sigmoid function.

[0092] For the classification loss function L cls And the positioning loss function L loc By performing linear combination, we can get the training loss function of labeled image samples and unlabeled image samples, that is:

[0093] l s or u =L cls +ηL loc (6)

[0094] Where η represents the weighting coefficient, which ranges from 0 to 1.

[0095] It should be noted that the loss part of the labeled image samples and the loss part of the unlabeled image samples with pseudo labels have the same form of classification loss and positioning loss, that is, Formula 6.

[0096] In the third part, combined with variance analysis, the positioning confidence is obtained based on the variance of the position offset confidence, and the normalized positioning confidence is combined with the classification confidence to construct a comprehensive confidence; the comprehensive confidence is used and the overall loss function of the category-weighted student network is introduced to alleviate the adverse effects of category imbalance on target detection.

[0097] The present invention designs a comprehensive confidence calculation method: first, according to the output value of the student network, the classification confidence and the position offset confidence are obtained. Figure 2 As shown in the dashed box, cxi, cyi, cai, cbi (i = 1, ..., n) represent the confidence of the position offset of the quaternion coordinates (x, y, a, b) in each grid in the i-th segment. The calculation formula for the positioning confidence is as follows:

[0098]

[0099]

[0100] where L′ conf represents the fixed-position reliability, D(i) represents the variance of variable i, and i in formula (8) j Indicates the confidence that the position offset i (i can be expressed as coordinates x, y, a, b) is located in the jth segment of the grid (for example, for the x coordinate, the value is Figure 2 The network output value cxi), represents the confidence mean of the i coordinate offset (i can be represented as coordinates x, y, a, b) (obtained by summing and averaging the confidence of each segment), and n represents the number of segments divided by the grid. The present invention adopts the normalized positioning confidence L conf , defined by formula (9):

[0101]

[0102] Where s represents the sigmoid function, and the function expression is

[0103] Next, the classification confidence and location confidence are combined to obtain the comprehensive confidence B score ,Right now:

[0104] B score =αcls score +(1-α)L conf (10)

[0105] where cls scorerepresents the classification confidence, which is obtained by the prediction output of the classification branch of the teacher network, and α represents the weighting coefficient, which ranges from 0 to 1. The comprehensive confidence calculated by formula (10) is used as a new standard for NMS processing and threshold filtering to obtain the pseudo label of the unlabeled image sample containing positioning and classification accuracy information.

[0106] Next, the overall loss function of the student network is weighted using the category-related comprehensive confidence, which can alleviate the impact of category imbalance on the semi-supervised target detection algorithm to a certain extent, as shown in formulas (11) and (12):

[0107]

[0108]

[0109] where c i represents a certain category subscript, and c represents all category subscripts; Represents the sum of the predicted comprehensive confidence of a certain category of unlabeled image samples in the batch data, Represents the sum of the predicted comprehensive confidence of the unlabeled image samples in the batch data; the comprehensive confidence related to the category Weighted with Formula 3, Formula (12) gives the weighted result of the overall loss function.

[0110] Part 4: Network Iteration Update Strategy

[0111] The semi-supervised target detection algorithm of the present invention adopts an end-to-end iterative training method. The student network is updated by reverse stochastic gradient descent according to the loss function, and the teacher network is updated by the EMA (exponential moving average) of the student network, that is:

[0112] θ′ t =αθ′ t-1 +(1-α)θ t (15)

[0113] where θ t represents the current model parameters of the student network, θ t ' -1 represents the student network parameter of the last iteration round, and α represents the weighting coefficient, which is generally set to 0.9. So far, the technical solution of the present invention has been explained.

[0114] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A semi-supervised object detection method for improving pseudo-label quality and class imbalance, characterized in that: include: Constructing a semi-supervised target detection model, the model comprising a teacher network and a student network based on the target detection network; The regression branch of the predicted offset in the target detection network is discretized, and the confidence of each offset segment is calculated, thus realizing the confidence of each segment of the predicted offset in the grid divided by the sample image; the loss function used to train the positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct the training loss function of the target detection network, thereby obtaining the overall loss function of the student network; Combined with variance analysis, the location confidence is obtained based on the variance of the location offset confidence, and the normalized location confidence is combined with the classification confidence to construct a comprehensive confidence. The overall loss function of the class-weighted student network is introduced by using the comprehensive confidence to alleviate the adverse effects of class imbalance on target detection. The student network is updated by reverse stochastic gradient descent according to the loss function, and the teacher network is updated by the EMA of the student network; When applied, for a picture to be detected, it is input into the trained student network, and the target detection result is obtained through the output of the student network; The combined variance analysis obtains the location confidence based on the variance of the offset, combines the normalized location confidence with the classification confidence, and constructs a comprehensive confidence, including: First, according to the output value of the student network, the classification confidence and position offset confidence are obtained; the calculation formula of the positioning confidence is as follows: where L′ conf represents the fixed-position reliability, D(i) represents the variance of variable i, i j represents the confidence that the i position offset is located in the jth segment in the grid, represents the confidence mean of the i-coordinate offset; n represents the number of segments of the grid; Using the normalized location reliability L conf , defined by formula (9): Where s represents the sigmoid function; Next, the classification confidence and location confidence are combined to obtain the comprehensive confidence B score ,Right now: B score =αcls score +(1-α)L conf (10) where cls score represents the classification confidence, which is obtained by the prediction output of the classification branch of the teacher network, and α represents the weighting coefficient, which ranges from 0 to 1; The comprehensive confidence calculated by formula (10) is used as a new standard for NMS processing and threshold filtering to obtain pseudo labels for unlabeled image samples containing positioning and classification accuracy information.

2. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The teacher network is trained: First, a batch of data is formed by sampling according to a preset ratio, and the unlabeled image samples in the batch of data are enhanced using weak data enhancement technology; Secondly, the unlabeled image samples are input into the randomly initialized target detection network, namely the teacher network, to obtain the prediction results of the unlabeled image samples. The prediction results are the classification confidence and position offset confidence of the target in the unlabeled image samples. The comprehensive confidence is obtained by combining the classification confidence and the positioning confidence. Thirdly, NMS processing is performed according to the comprehensive confidence, and a threshold τ is set to filter the bounding boxes after NMS processing, that is, the prediction results of the bounding boxes after NMS processing whose comprehensive confidence is greater than the threshold are retained.

3. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The student network is trained: Weak data enhancement is performed on labeled image samples in batch data, and strong data enhancement is performed on unlabeled image samples with pseudo labels; the strong enhancement is a Mixup operation, which mixes unlabeled image samples with pseudo labels with labeled image samples in proportion to form an enhanced data set; then, the enhanced data set is input into a target detection network with the same random initialization parameters as the aforementioned teacher network, namely, a student network; the student network is then updated by performing stochastic gradient descent on the overall loss function weighted by the comprehensive confidence, and the teacher network is updated by the exponential moving average EMA of the student network; when the student network converges, the training is completed and it is used alone for target detection in images.

4. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The step of discretizing the regression branch of the predicted offset in the target detection network and calculating the confidence of each segment of the offset realizes the confidence of each segment of the predicted offset in the grid divided by the sample image, including: The object detection network adopts Faster R-CNN; based on Faster R-CNN, the prediction offset regression head part is discretized and improved, and the offset regression head is converted into a classification head; that is, the labeled image samples / unlabeled image samples are divided into multiple grids, each grid is divided into multiple segments, and the median of each segment is taken as the approximate value of the discrete offset in the segment, and the confidence of the offset of each segment is reflected as the probability that the offset is located in the segment.

5. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The image samples in the enhanced dataset are input into the student network. After being processed and output by the RPN region extraction network in the student network, the upper left coordinates (x, y) and lower right coordinates (a, b) of the real bounding box and the pseudo-label bounding box are calculated online according to formulas (1) and (2), corresponding to the offset of the filtered bounding box output by the RPN region extraction network: Among them G x , G y are the horizontal and vertical coordinates of the true bounding box or the horizontal and vertical coordinates of the pseudo-label bounding box, respectively. x , P y are the horizontal and vertical coordinates of the bounding box after filtering output by the corresponding RPN network, P w , P h The width and height of the filtered bounding box output by the RPN network; The offset calculation result is downsampled and returned to the coordinate system of the last layer output feature map of Faster R-CNN. This downsampled result is compared with the grid to which the filtered bounding box output by the RPN network belongs. Information 1 is recorded at the corresponding position and information 0 is recorded in other places, forming a one-hot encoding of the position offset information, which is used for subsequent training to calculate the positioning loss.

6. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The loss function used for training positioning accuracy in the student network is transformed into a binary cross entropy classification loss function, and linearly combined with the classification loss to construct a training loss function for the target detection network, thereby obtaining the overall loss function of the student network, including: The overall loss function calculation of the student network is represented by the standard semi-supervised loss function, and the overall loss function is defined as shown in formula (3): loss=l s +λl u (3) Among them l s Represents the loss part of the labeled image sample, l u represents the loss part of the unlabeled image sample with pseudo labels, λ represents the weighting coefficient, and takes values ​​in the closed interval [1,2]; l s and l u The classification and positioning loss calculation formulas have the same form, as shown in Equation 4 and Equation 5: Where L cls represents the classification loss, L loc represents the positioning loss; x, y represent the horizontal and vertical coordinates of the upper left corner of the bounding box, a, b represent the horizontal and vertical coordinates of the lower right corner of the bounding box; loc represents the subscripts of each segment divided by each grid, cls represents the category subscript; pos represents the subscript of the bounding box in the filtered bounding box output by the RPN network whose IOU with the real bounding box and the pseudo-label bounding box is greater than 0.7, N pos Indicates the number of bounding boxes with subscript pos; B ij With B ijk The value of is {0,1}, where 1 means existence and 0 means non-existence; When calculating the positioning loss, B ijk = 1 indicates whether the offset of the j-coordinate of the i-th bounding box is located in the k-th segment in the discretized grid; when calculating the classification loss, B ij =1 means that there is a target of the jth type in the i-th bounding box; P ij represents the category score of the jth category of the i-th bounding box output by the student network, and this is used as the classification confidence; P ijk represents the confidence of the offset of the j-coordinate in the i-th bounding box in the k-th segment, and s(■) represents the sigmoid function; For the classification loss function L cls And the positioning loss function L loc By performing linear combination, we can get the training loss function of labeled image samples and unlabeled image samples, that is: l s or u =L cls +ηL loc (6) Where η represents the weighting coefficient, which ranges from 0 to 1.

7. The semi-supervised target detection method for improving pseudo-label quality and class imbalance according to claim 1, characterized in that: The overall loss function of the class-weighted student network using comprehensive confidence is introduced to alleviate the adverse effects of class imbalance on target detection, including: where c i represents a certain category subscript, and c represents all category subscripts; Represents the sum of the predicted comprehensive confidence of a certain category of unlabeled image samples, Represents the sum of the predicted comprehensive confidence of the unlabeled image samples; the category-related comprehensive confidence β ci Weighted with the overall loss function, formula (12) gives the weighted result of the overall loss function.

Images