A method, system, device and medium for class-imbalanced semi-supervised target detection

Abstract

This invention provides a method, system, device, and medium for semi-supervised object detection in imbalanced image scenarios. The method involves inputting an acquired imbalanced image dataset into a pre-defined object detector for object detection analysis. This analysis yields classification losses for unlabeled and unlabeled images, as well as pseudo-labels for unlabeled images. Gradient matrices are calculated to establish a set of class gradient balance equations. The total classification loss is calculated based on the class weights and weight loss obtained from solving these equations. During iterative decision-making, effective pseudo-labels and image sampling rates for unlabeled images are determined based on label class thresholds obtained from the class weights. The pre-defined object detector is then trained using the effective pseudo-labels, image sampling rates, and total classification loss to obtain a semi-supervised object detector. This detector is then used to perform object detection on the images to be detected, yielding the detection results. This invention effectively ensures the accuracy of object detection in imbalanced scenarios and improves the application performance of semi-supervised object detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a class-imbalanced semi-supervised target detection method, system, computer equipment, and storage medium. Background Technology

[0002] In natural scenes, different types of objects appear with varying frequencies, inevitably leading to class imbalance in the datasets used for object detection. Semi-supervised object detection (SSOD) methods utilize both labeled and unlabeled data to improve object detector performance; therefore, their performance is highly susceptible to class imbalance, especially in minority class detection scenarios. Consequently, addressing the class imbalance problem in SSOD has become a key challenge for the application of semi-supervised object detection.

[0003] Existing object detectors for addressing class imbalance mainly fall into two categories: those based on supervised learning of class balance and those that mitigate class imbalance by using Focal loss or applying class-level thresholding strategies. Among these, supervised learning-based class balance detectors, such as Long Tail Object Detection (LTOD) and Few-Shot Object Detection (FSOD), are designed based on a supervised paradigm. They cannot fully utilize the latent information in unlabeled data, are prone to overfitting limited labeled images of minority classes, and are difficult to integrate with semi-supervised object detection (SSOD) methods. Furthermore, SSOD methods that mitigate class imbalance by using Focal loss or applying class-level thresholding strategies are susceptible to two types of confirmation bias: majority-class biased models tend to predict pseudo-labels that are also biased towards the majority class, leading to overfitting of biased pseudo-labels, and incorrect minority-class pseudo-labels dominating training and causing classification bias. This prevents them from generating high-quality pseudo-labels for the minority class in highly imbalanced semi-supervised datasets, thus affecting object detection performance. Summary of the Invention

[0004] The purpose of this invention is to provide a semi-supervised object detection method for class imbalance. By combining a gradient-based reweighting mechanism that estimates class weights from the gradient matrix to balance the positive and negative gradients of the classes with a gradient-based dynamic adaptive thresholding strategy and a class-balanced resampling strategy, this method addresses the application shortcomings of existing semi-supervised object detectors. It can effectively solve the two-class confirmation error in class imbalance scenarios, and reliably detect the majority and minority classes using unlabeled data, ensuring the accuracy of object detection in class imbalance scenarios and improving the application performance of semi-supervised object detection.

[0005] To achieve the above objectives, it is necessary to provide a class-imbalanced semi-supervised target detection method, system, computer equipment, and storage medium to address the aforementioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a class-imbalanced semi-supervised target detection method, the method comprising the following steps:

[0007] A class-imbalanced image dataset is acquired and input into a preset target detector for target detection analysis to obtain unlabeled image classification loss, labeled image classification loss, and pseudo-labels for each unlabeled image. The class-imbalanced image dataset includes a labeled image subset and an unlabeled image subset. The pseudo-labels include label category, category logical value, and detection box position coordinates.

[0008] Calculate the partial derivative of the category image classification loss for each label category with respect to the logistic value of each category to obtain the corresponding gradient matrix, and establish the corresponding category gradient balance equation system based on the gradient matrix; the gradient matrix includes the positive gradient and negative gradient of each label category;

[0009] Solve the class gradient balance equations to obtain the class weights and weight loss for each label class, and obtain the total classification loss based on the class weights and weight loss for each label class.

[0010] Based on the category weights of each label category, the corresponding label category thresholds are obtained. Based on the label category thresholds, the effective pseudo-labels and the image sampling rates of each unlabeled image are obtained. Based on the effective pseudo-labels, the sampling rates of each image, and the total classification loss, the preset target detector is trained and updated to obtain a semi-supervised target detector.

[0011] The image to be detected is acquired, and the image is subjected to target detection based on the semi-supervised target detector to obtain the detection result.

[0012] Further, the step of calculating the partial derivatives of the category image classification loss for each label category with respect to the logistic values ​​of each category to obtain the corresponding gradient matrix includes:

[0013] Based on the unlabeled image classification loss and the labeled image classification loss for each label category, the positive gradient and negative gradient for each label category are calculated respectively; the positive gradient is expressed as:

[0014]

[0015] in, Indicates label category The positive gradient; Indicates the first The logical value of the category of each candidate box; Represents category logical value The Middle Tag categories The corresponding logical value score; and These represent the loss function for classified images with and without labels, respectively.

[0016] The negative gradient is represented as:

[0017]

[0018] in, Indicates the label category in the same batch For tag categories The negative gradient, and ;

[0019] Based on the positive and negative gradients of each label category, the corresponding batch learning gradient matrix is ​​obtained;

[0020] The gradient matrix is ​​obtained by performing a moving average calculation on the batch learning gradient matrix; the gradient matrix is ​​represented as follows:

[0021]

[0022] in, Represents the batch learning gradient matrix; Represents the gradient matrix; This represents the moving average coefficient of the gradient matrix.

[0023] Further, the step of establishing the corresponding class gradient balance equation system based on the gradient matrix includes:

[0024] Based on the principle that the positive and negative gradients of each label category are equal, the gradient balance equations for each label category are obtained according to the gradient matrix; the gradient balance equations are expressed as follows:

[0025]

[0026] in, and They represent the gradient matrices respectively. Chinese label category Positive gradient and label category For tag categories The negative gradient; and Representing the tag categories and tag categories Corresponding category weights;

[0027] Based on the gradient balance equations for all label categories and the category weight constraint equations, the set of category gradient balance equations is obtained; the category weight constraint equations are expressed as follows:

[0028]

[0029] in, Indicates label category The corresponding category weights; n represents the total number of categories.

[0030] Furthermore, the step of solving the class gradient balance equations to obtain the class weights and weight loss for each label class includes:

[0031] Based on the preset learnable category logic, a weight estimation function is generated; the weight estimation function is expressed as:

[0032]

[0033] in, This represents the weight of the i-th category; n represents the total number of categories; Indicates and The corresponding learnable category logic; Indicates and The corresponding learnable category logic;

[0034] Based on the weight estimation function, the Jacobian iteration process is used to iteratively solve the class gradient balance equations to obtain the learnable class logic corresponding to each class weight; the Jacobian iteration process is expressed as follows:

[0035]

[0036]

[0037] in, and They represent the first The target weight corresponding to the i-th category weight in the round of iteration;

[0038] By combining the learnable category logic corresponding to each category weight with the weight estimation function, we obtain the corresponding category weights and weight calculation loss; the weight calculation loss is expressed as:

[0039]

[0040] in, This indicates the weighted loss value.

[0041] Furthermore, the step of calculating the total classification loss based on the category weights of each label category and the weights includes:

[0042] The class weights of each label category corresponding to the labeled image subset are smoothed to obtain the corresponding labeled class weights; the labeled class weights are expressed as follows:

[0043]

[0044] in, and These represent the weight of the i-th class and the corresponding labeled class weight, respectively, for the labeled image subset. Indicates hyperparameters;

[0045] The image classification loss is obtained based on the weights of each labeled category and the weights of each label category corresponding to the labeled image subset.

[0046] The total classification loss is obtained by summing the image classification loss and the weight solving loss; the total classification loss is expressed as:

[0047]

[0048] In the formula,

[0049]

[0050] in, , and These represent the total classification loss, the image classification loss, and the weight calculation loss, respectively. This indicates that there are labeled category weights.

[0051] Furthermore, the step of obtaining the corresponding tag category threshold based on the category weight of each tag category includes:

[0052] Based on a gradient adaptive strategy, the corresponding label category threshold is obtained according to the category weight of each label category; the label category threshold is expressed as:

[0053]

[0054] in, This is a preset threshold hyperparameter; The label category threshold representing the i-th label category; This represents the category weight of the i-th label category.

[0055] Furthermore, the step of obtaining the effective pseudo-labels and the image sampling rates of each unlabeled image based on the threshold values ​​of each label category includes:

[0056] The pseudo-labels of each unlabeled image are filtered according to the label category threshold to obtain the effective pseudo-labels;

[0057] Based on the category logical value of each valid pseudo-label and the corresponding label category threshold, the corresponding unlabeled category resampling rate is obtained; the unlabeled category resampling rate is expressed as:

[0058]

[0059] In the formula,

[0060]

[0061]

[0062] in, Indicates label category The corresponding unlabeled category resampling rate; and These represent the tag category and category logical value of a valid pseudo-tag, respectively; Indicates label category The label category threshold; A coefficient indicating the range of the adjusted sampling rate; Indicates the number of training iterations; Indicates the current training iteration round number; This indicates the preset dynamic sampling threshold; and These represent the number of labeled images and the number of unlabeled images in the class-imbalanced image dataset, respectively. This indicates that the class-imbalanced image dataset contains at least one labeled class. The number of images marked in the box;

[0063] The maximum value of the unlabeled category resampling rate for each unlabeled image is selected as the corresponding image sampling rate.

[0064] Secondly, embodiments of the present invention provide a class-imbalanced semi-supervised target detection system, the system comprising:

[0065] The pseudo-label generation module is used to acquire an imbalanced image dataset and input the imbalanced image dataset into a preset target detector for target detection analysis to obtain unlabeled image classification loss, labeled image classification loss, and pseudo-labels for each unlabeled image; the imbalanced image dataset includes a labeled image subset and an unlabeled image subset; the pseudo-label includes label category, category logical value, and detection box position coordinates;

[0066] The gradient balancing modeling module is used to calculate the partial derivative of the category image classification loss of each label category with respect to the logistic value of each category, obtain the corresponding gradient matrix, and establish the corresponding category gradient balancing equation system based on the gradient matrix; the gradient matrix includes the positive gradient and negative gradient of each label category;

[0067] The classification loss calculation module is used to solve the class gradient balance equation system to obtain the class weights and weight-based loss of each label class, and to obtain the total classification loss based on the class weights and weight-based loss of each label class.

[0068] The resampling training module is used to obtain the corresponding label category threshold according to the category weight of each label category, and to obtain the effective pseudo-label and the image sampling rate of each unlabeled image according to each label category threshold. The module is also used to train and update the preset target detector according to the effective pseudo-label, the sampling rate of each image, and the total classification loss to obtain a semi-supervised target detector.

[0069] The target detection module is used to acquire the image to be detected and perform target detection on the image to be detected according to the semi-supervised target detector to obtain the detection result.

[0070] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0071] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0072] This application provides a method, system, computer device, and storage medium for class-imbalanced semi-supervised object detection. The method acquires a class-imbalanced image dataset comprising labeled and unlabeled image subsets, inputs the dataset into a preset object detector for object detection analysis, and obtains the unlabeled image classification loss, the labeled image classification loss, and pseudo-labels for each unlabeled image. Then, it calculates the partial derivatives of the class image classification loss for each labeled class with respect to the logistic values ​​of each class, obtaining gradient matrices including the positive and negative gradients for each labeled class. Based on these gradient matrices, a corresponding [system / mechanism] is established. After balancing the class gradient equations, the class weights and weight loss for each label class are obtained by solving the class gradient balancing equations. Then, the total classification loss is obtained based on the class weights and weight loss of each label class. Next, the corresponding label class thresholds are obtained based on the class weights of each label class. Finally, the effective pseudo-labels and image sampling rates for each unlabeled image are obtained based on the label class thresholds. Based on the effective pseudo-labels, image sampling rates, and total classification loss, a pre-defined semi-supervised object detector is resampled, trained, and updated to obtain the object detector. Object detection is then performed on the image to be detected using the object detector to obtain the detection result. Compared with existing technologies, this class-imbalanced semi-supervised object detection method effectively solves the two-class confirmation error in class-imbalanced scenarios by combining a gradient-based reweighting mechanism (estimating class weights from the gradient matrix to balance positive and negative gradients) with a gradient-based dynamic adaptive threshold strategy and a class-balanced resampling strategy. It utilizes unlabeled data to reliably detect both the majority and minority classes, ensuring the accuracy of object detection in class-imbalanced scenarios and improving the application performance of semi-supervised object detection. Attached Figure Description

[0073] Figure 1 This is a schematic diagram illustrating an application scenario of the class-imbalanced semi-supervised target detection method in this embodiment of the invention;

[0074] Figure 2 This is a schematic diagram of the process framework for class-imbalanced semi-supervised target detection in an embodiment of the present invention;

[0075] Figure 3 This is a flowchart illustrating the class-imbalanced semi-supervised target detection method in an embodiment of the present invention;

[0076] Figure 4 This is a schematic diagram illustrating the process of obtaining the gradient matrix and solving the class gradient balance equation system in an embodiment of the present invention;

[0077] Figure 5 This is a schematic diagram of the structure of the class-imbalanced semi-supervised target detection system in an embodiment of the present invention;

[0078] Figure 6This is an internal structural diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and beneficial effects of this application clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described below are only part of the embodiments of the present invention and are used to illustrate the present invention, but are not intended to limit the scope of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0080] The class imbalance semi-supervised object detection method provided in this invention can be understood as addressing the performance limitations of existing semi-supervised object detectors, which fail to effectively solve two types of confirmation errors: models biased towards the majority class tend to predict pseudo-labels that are also biased towards the majority class, leading to overfitting of biased pseudo-labels; and incorrect minority class pseudo-labels easily dominate training, causing classification bias. This invention proposes a method that fully balances the positive and negative gradients of classes by estimating class weights from the gradient matrix. This matrix is ​​used as an indicator of the model's training state to set optimization goals for class weights, mitigating confirmation bias towards the majority class. Furthermore, it designs dynamic thresholds for unlabeled data based on gradient adaptive strategies and performs class-balanced resampling training based on pseudo-label confidence and dynamic class thresholds. This drives the model to learn from high-confidence pseudo-labels, reducing noise introduced by erroneous pseudo-labels. This semi-supervised object detection scheme can generate high-quality pseudo-labels for the minority class and can be applied to applications such as... Figure 1 The terminal or server shown. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The server can be implemented using the features provided by this invention, depending on the actual application requirements. Figure 2 The class-imbalanced semi-supervised object detection framework shown performs efficient and accurate object detection, and the obtained object detection results are used for subsequent research on the server or transmitted to the terminal for end users to view and analyze; the following embodiments will provide a detailed description of the class-imbalanced semi-supervised object detection method of the present invention.

[0081] In one embodiment, Figure 3 As shown, a class-imbalanced semi-supervised object detection method is provided, including the following steps:

[0082] S11. Obtain the class-imbalanced image dataset and input it into a preset target detector for target detection analysis to obtain the unlabeled image classification loss, the labeled image classification loss, and the pseudo-labels for each unlabeled image; the class-imbalanced image dataset includes a labeled image subset. and unlabeled image subsets ,and It can be understood as a having A list of bounding boxes labeled with category labels. and detection box position coordinates composition, and These represent the number of labeled and unlabeled images, respectively; furthermore, in this embodiment, all categories are denoted as... and This represents the total number of categories, and the corresponding pseudo-labels include the label category, category logical value, and detection box position coordinates.

[0083] The model structure of the aforementioned preset object detector can be selected according to actual application needs. For example, object detection models based on the student-teacher framework or the YOLO series architecture can be used based on the acquired labeled image subset dataset. and unlabeled image subsets The following methods and steps are used to train a semi-supervised object detector with optimal parameters that can perform well in both majority and minority class object detection. No specific limitations are made here.

[0084] S12. Calculate the partial derivative of the category image classification loss for each label category with respect to the logistic value of each category to obtain the corresponding gradient matrix. Based on the gradient matrix, establish the corresponding category gradient balance equation system. The gradient matrix can be understood as a matrix obtained based on the mandatory principle that the positive and negative gradients of each label category are equal. It is used as an indicator to measure the model's training state and to formulate the optimization objective for class weights. This aims to estimate the class weights to balance the positive and negative gradients of each category, thereby reducing the confirmation bias for most classes and making the predicted probability value of each category tend to stabilize during training. Specifically, it includes the matrices of the positive and negative gradients of each label category. Figure 4 As shown, the step of calculating the partial derivatives of the category image classification loss for each label category with respect to the logistic values ​​of each category to obtain the corresponding gradient matrix includes:

[0085] Based on the unlabeled image classification loss and labeled image classification loss for each label category, the positive and negative gradients for each label category are calculated respectively; whereby the unlabeled image classification loss can be understood as the classification loss of the pre-defined object detector on the unlabeled image dataset, denoted as . The labeled image classification loss can be understood as the classification loss of the pre-defined object detector on the labeled image dataset, denoted as . It should be noted that in this embodiment, the loss functions for calculating the classification loss of unlabeled images and the classification loss of labeled images can both be loss functions adapted to the model structure corresponding to the preset target detector, and no specific limitation is made here.

[0086] In practical object detection training tasks, the gradient that increases the prediction probability (logistic value) of the last layer for each class is represented as a positive gradient, and the gradient that decreases the prediction probability (logistic value) of that class is represented as a negative gradient. When the positive gradient of a class exceeds its negative gradient, the average score of that class tends to become larger, and the model's final prediction may be biased towards that class, causing an imbalance in unsupervised classification training. Simultaneously, due to domain differences and class imbalance, the majority and easy classes often have more pseudo-labels, which also leads to the positive gradients of the majority and easy classes often being greater than their negative gradients, while the gradients of other classes are the opposite. Because existing methods cannot completely balance the positive and negative gradients of each class, this gradient imbalance leads to training on unlabeled data using pseudo-labels biased towards the majority class, thereby increasing the score of the majority class prediction and causing the model to resist the prediction of the minority class, resulting in class imbalance. To address class identification bias and class imbalance, this embodiment preferably calculates the positive and negative gradients of each class to reweight them. and This ensures that the positive and negative gradients are equal for each category.

[0087] Suppose that in a batch, the first The logical category value of each candidate box in the category header is... And the corresponding pseudo-tag is and in The Middle One channel Representing the first kind The corresponding logical value score; and These are the classification loss functions for labeled and unlabeled data, respectively, and during this batch of learning, and represents the number of candidate bounding boxes in the labeled and unlabeled images, respectively. Therefore, based on the theory of category gradient calculation, Positive gradient of class The tag category in the pseudo-tag can be calculated as follows: All samples (for The loss for positive samples is calculated based on the logic value score (logic value). The partial derivatives yield the positive gradient as follows:

[0088]

[0089] in, Indicates label category The positive gradient; Indicates the first The logical value of the category of each candidate box; Represents category logical value The Middle Tag categories The corresponding logical value score; and These represent the loss function for classified images with and without labels, respectively.

[0090] Similarly, by calculating the tag category in the pseudo-tags... ( The partial derivatives of all samples can be obtained from the samples from category ) within a batch. For categories negative gradient Represented as:

[0091]

[0092] in, Indicates the label category in the same batch For tag categories The negative gradient, and ;

[0093] It should be noted that when there is no true label in a certain training batch... When selecting candidate boxes, for All ;

[0094] Based on the positive and negative gradients of each label category, the corresponding batch learning gradient matrix is ​​obtained; where, the i-th gradient in the batch learning gradient matrix... row and number The matrix element values ​​of the column are calculated above. And the first The row represents the number of candidate box samples from each class to... The gradient, the first Columns represent from The gradient from the candidate bounding boxes to each class;

[0095] In practical applications, the batch learning gradient matrix obtained above can be directly used to solve the class weights. However, in order to obtain more stable calculation results of gradient statistics, this embodiment preferably estimates the final gradient matrix by sliding the gradient obtained from a single batch using the following method.

[0096] The gradient matrix is ​​obtained by performing a moving average calculation on the batch learning gradient matrix; the gradient matrix is ​​represented as follows:

[0097]

[0098] in, Represents the batch learning gradient matrix; Represents the gradient matrix; This represents the moving average coefficient of the gradient matrix.

[0099] As mentioned above, an imbalance between positive and negative gradients for each class can lead to biased model predictions. To balance the positive and negative gradients for each class, this embodiment preferably uses a set of learnable class loss weights to reweight the segmentation loss. For example, for class... The loss weight is defined as Furthermore, for each foreground class, it is expected that its positive gradient and negative gradient are equal, and a corresponding class gradient balance equation system is established accordingly; specifically, the step of establishing the corresponding class gradient balance equation system based on the gradient matrix includes:

[0100] Based on the principle that the positive and negative gradients of each label category are equal, the gradient balance equations for each label category are obtained according to the gradient matrix; the gradient balance equations are expressed as follows:

[0101]

[0102] in, and They represent the gradient matrices respectively. Chinese label category Positive gradient and label category For tag categories The negative gradient; and Representing the tag categories and tag categories Corresponding category weights;

[0103] As shown above, we can obtain Article about The gradient balance equation is not detailed here;

[0104] Based on the gradient balance equations for all label categories and the category weight constraint equations, the category gradient balance equation set is obtained. The category weight constraint equation can be understood as an additional weight constraint introduced to fix the range of each category weight, and the (n+1)th gradient balance equation representing the background class is replaced accordingly to obtain the category gradient balance equation set that can be used to solve for the category weights. Specifically, the category weight constraint equation is expressed as follows:

[0105]

[0106] in, Indicates label category The corresponding category weights; n represents the total number of categories.

[0107] After obtaining the class gradient balancing equations based on the above methods and steps, the class weights can be obtained by optimizing these equations. The class weights are then used to reweight the classification loss to achieve class balance during training.

[0108] S13. Solve the category gradient balance equations to obtain the category weights and weight loss for each label category, and obtain the total classification loss based on the category weights and weight loss for each label category. The category gradient balance equations can theoretically be solved using conventional linear methods of matrix inversion and matrix multiplication. However, considering that this method is feasible with a small number of categories, solving the gradient equations is often unstable with a large number of categories, easily resulting in negative values ​​or even no solution. Therefore, this embodiment preferably uses an optimizer to simulate the Jacobian iteration method to solve the category gradient balance equations. Specifically, as shown... Figure 4 As shown, the steps for solving the class gradient balance equations to obtain the class weights and weight loss for each label class include:

[0109] Based on the pre-defined learnable category logic, a weight estimation function is generated; whereby the learnable category logic can be understood as a set of learnable category logics established to simulate the Jacobian iterative training of weights. Used to estimate weights using functions The activation value; the corresponding weight estimation function is preferably the softmax function, to ensure The value is a positive number. Specifically, the weight estimation function is expressed as:

[0110]

[0111] in, This represents the weight of the i-th category; n represents the total number of categories; Indicates and The corresponding learnable category logic; Indicates and The corresponding learnable category logic; it should be noted that these activation values ​​are... All were initialized to 0;

[0112] Based on the weight estimation function, the Jacobi iteration process is used to iteratively solve the class gradient balance equations to obtain the learnable class logic corresponding to each class weight. It should be noted that, for ease of description of the solution process, the Jacobi iteration process will be explained below using the t-th training round as an example: The training round... The weights of the wheels are all denoted as Then update This causes the weights to shift towards the set target weights. Approaching, and target weight This can be calculated using the Jacobi iteration method. Specifically, for the class gradient balance equation system, one round of Jacobi iteration can be represented as:

[0113]

[0114] And for the background class, we have:

[0115]

[0116] in, and They represent the first The target weight corresponding to the i-th category weight in the round of iteration;

[0117] By combining the learnable category logic corresponding to each category weight with the weight estimation function, the corresponding category weights and weight calculation loss are obtained; wherein, the learnable category logic... Once obtained through the Jacobi iteration process, it can be substituted into the corresponding weight estimation function to calculate the required class weights.

[0118] It should be noted that regularization is applied in each round of Jacobi iteration. Alignment and This can be achieved by minimizing the loss value, which is defined as the weighted solution loss, expressed as:

[0119]

[0120] in, This indicates the weighted solution for the loss value;

[0121] To ensure the reliability of subsequent weighted calculations of labeled and unlabeled image classification losses based on category weights, and to effectively prevent the loss term from being dominated by large weights, thus avoiding the risk of small weights failing to be optimized, thereby improving the effectiveness of detector model training, this embodiment preferably incorporates the weight-based loss calculation when calculating the total training classification loss, and uses it in conjunction with the labeled and unlabeled image classification losses weighted by category weights; specifically, the step of obtaining the total classification loss based on the category weights of each label category and the weight-based loss calculation includes:

[0122] The class weights of each label category corresponding to the labeled image subset are smoothed to obtain the corresponding labeled class weights. The smoothing process can be understood as a simple exponentiation operation to prevent the model from overfitting. The corresponding labeled class weights are expressed as follows:

[0123]

[0124] in, and These represent the weight of the i-th class and the corresponding labeled class weight, respectively, for the labeled image subset. Indicates hyperparameters;

[0125] The image classification loss is obtained based on the weights of each labeled category and the weights of each label category corresponding to the labeled image subset.

[0126] The total classification loss is obtained by summing the image classification loss and the weight solving loss; the total classification loss is expressed as:

[0127]

[0128] In the formula,

[0129]

[0130] in, , and These represent the total classification loss, the image classification loss, and the weight calculation loss, respectively. This indicates that there are labeled category weights;

[0131] This embodiment uses a gradient-based reweighting optimization scheme for the classification loss, which can effectively alleviate class imbalance from the perspective of gradient level. This makes the logistic values ​​of all classes tend to be balanced and stable during training, thereby eliminating the confirmation bias caused by overfitting biased pseudo-labels due to the majority class bias model tending to predict pseudo-labels that are also biased towards the majority class. This effectively reduces the target detection bias.

[0132] S14. Based on the category weights of each label category, obtain the corresponding label category thresholds, and based on each label category threshold, obtain the effective pseudo-labels and the image sampling rates of each unlabeled image. Based on the effective pseudo-labels, the sampling rates of each image, and the total classification loss, train and update the preset target detector to obtain a semi-supervised target detector.

[0133] Existing object detectors do not consider the complexity and class distribution of different samples during training. They often use fixed thresholds and thresholds determined based on class scores for pseudo-label filtering, which cannot obtain more pseudo-labels for the minority class and is not conducive to alleviating the class imbalance problem. In order to alleviate the shortcomings of existing class threshold applications and effectively prevent the detector from overfitting to the few pseudo-labels of the minority class (especially when the class weight of the minority class is large), this embodiment preferably uses an adaptive thresholding strategy based on gradients to determine the class threshold. Specifically, the step of obtaining the corresponding label class threshold according to the class weight of each label class includes:

[0134] Based on a gradient adaptive strategy, the corresponding label category threshold is obtained according to the category weight of each label category; the label category threshold is expressed as:

[0135]

[0136] in, This is a preset threshold hyperparameter; The label category threshold representing the i-th label category; This represents the category weight of the i-th label category;

[0137] Based on the expression for the above label category threshold, it is easy to see that for Higher classes, A smaller value allows the target detector to adaptively acquire more pseudo-labels for the minority classes, thereby mitigating the class imbalance problem by addressing the first type of confirmation bias. Furthermore, as a supplement to the gradient-based reweighting mechanism that balances the positive and negative gradients of the classes by estimating class weights from the gradient matrix, it can effectively prevent the detector from overfitting to the few pseudo-labels of the minority classes.

[0138] Furthermore, considering that in practical applications, incorrect minority class pseudo-labels may easily dominate training, causing the detector to adapt to the incorrect information of the pseudo-labels and resulting in confirmation bias in classification, in order to mitigate the impact of incorrect pseudo-labels that cause this confirmation bias and improve the learning performance of the minority class detector, this embodiment preferably employs a class-balanced resampling mechanism to force the model to learn high-confidence pseudo-labels to reduce the interference of erroneous pseudo-labels; specifically, the step of obtaining the effective pseudo-labels and the image sampling rates of each unlabeled image based on the threshold of each label class includes:

[0139] The pseudo-labels of each unlabeled image are filtered according to the label category threshold to obtain the effective pseudo-labels;

[0140] Based on the category logical value of each valid pseudo-label and the corresponding label category threshold, the corresponding unlabeled category resampling rate is obtained; whereby, the unlabeled category resampling rate can be understood as the sampling rate of each category bounding box in the unlabeled image obtained based on probability score, category frequency of unlabeled pseudo-labels and dynamic label category threshold estimation based on category weight;

[0141] To facilitate understanding of the unlabeled category resampling rate estimation method, the following section will focus on the first... The dynamic class resampling strategy generated in the second generation will be explained in detail: Assume that the... The pseudo-labels are generated with unlabeled data containing pseudo-labels. for ,exist It contains at least one category The number of images marked in the box is Furthermore, it is desirable that the sampling process follows the principle of a smaller sampling rate for the minority class and a larger sampling rate for the majority class. This embodiment adopts a method similar to Repeat Factor Sampling (RFS), which samples each class... The resampling rate is defined as:

[0142]

[0143] in, It is a dynamic threshold that gradually increases during the training process to learn a better representation; A coefficient indicating the range of the adjusted sampling rate; This indicates the number of training rounds, i.e., the total number of generation processes; Indicates the current training round number; and These represent the number of labeled images and the number of unlabeled images in the class-imbalanced image dataset, respectively.

[0144] Suppose an unlabeled image The pseudo-tags are ,in , and These are the pseudo-label's label category, category logical value, and detection box position coordinates, respectively. yes The number of pseudo-labels differs from the sampling rate in RFS in that for each pseudo-label... , and The sampling rate and the number of image repetitions are calculated based on the pseudo-label scores. The unlabeled category resampling rate is expressed as:

[0145]

[0146] In the formula,

[0147]

[0148]

[0149] in, Indicates label category The corresponding unlabeled category resampling rate; Indicates label category The label category threshold; This indicates that the class-imbalanced image dataset contains at least one labeled class. The number of images marked in the box;

[0150] The maximum value of the unlabeled category resampling rate for each unlabeled image is selected as the corresponding image sampling rate; whereby, the image sampling rate can be understood as the frequency of sampling any unlabeled image in the unlabeled image data, i.e., the image sampling rate. The sampling rate is defined as The maximum value in;

[0151] It should be noted that in this embodiment, the repeated factor sampling (RFS) mechanism can be used to resample the labeled image dataset. By combining it with the sampling method for unlabeled images, the model can tend to learn high-confidence pseudo-labels to reduce the interference of erroneous pseudo-labels, especially those of minority classes, thereby mitigating the confirmation bias caused by the use of incorrect pseudo-labels. At the same time, the sampling rate of unlabeled images based on the label category threshold can smooth the positive and negative gradients of each class during training by sampling classes with small weights more times. This complements the gradient-based reweighting mechanism and is more beneficial for detecting and recognizing classes with simpler samples and lower frequency of occurrence in unlabeled images, thus effectively ensuring good detection performance in both majority and minority classes.

[0152] The simple and effective gradient-based CI-SSOD sampling training framework described above provides a semi-supervised object detector tailored for class-imbalanced semi-supervised object detection, which can be directly used for subsequent object detection analysis of similar images.

[0153] S15. Obtain the image to be detected, and perform target detection on the image to be detected according to the semi-supervised target detector to obtain the detection result; wherein, the image to be detected can be understood as the image that actually needs to be analyzed for target detection. The specific process of using the semi-supervised target detector to perform detection and analysis on the image to be detected can be referred to the relevant description of the training process of the semi-supervised target detector mentioned above, which will not be repeated here.

[0154] This application embodiment obtains a class-imbalanced image dataset including labeled and unlabeled image subsets, and inputs the class-imbalanced image dataset into a preset object detector for object detection analysis. This yields the unlabeled image classification loss, the labeled image classification loss, and the pseudo-labels for each unlabeled image. Then, it calculates the partial derivatives of the class image classification loss for each labeled category with respect to the logistic values ​​of each category, obtaining gradient matrices including the positive and negative gradients for each labeled category. Based on the gradient matrices, it establishes a corresponding class gradient balance equation system, solves the class gradient balance equation system to obtain the class weights and weight-based loss for each labeled category, and finally obtains the total classification loss based on the class weights and weight-based loss for each labeled category. When it is determined whether further iterative learning is needed based on the total classification loss, it then... The scheme involves obtaining the corresponding label category threshold by calculating the category weights of each label category, obtaining the effective pseudo-labels and the image sampling rate of each unlabeled image based on each label category threshold, and resampling and training the preset semi-supervised object detector based on the effective pseudo-labels and each image sampling rate to obtain the object detector. The object detector is then used to perform object detection on the image to be detected to obtain the detection result. This scheme effectively solves the two-class confirmation error in class imbalance scenarios by combining the gradient-based reweighting mechanism that balances the positive and negative gradients of the class by estimating the class weights from the gradient matrix with the gradient-based dynamic adaptive threshold strategy and the class balance resampling strategy. It uses unlabeled data to perform reliable object detection for the majority and minority classes, ensuring the accuracy of object detection in class imbalance scenarios and improving the application performance of semi-supervised object detection.

[0155] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0156] In one embodiment, Figure 5 As shown, a class-imbalanced semi-supervised target detection system is provided, the system comprising:

[0157] The pseudo-label generation module 1 is used to acquire an imbalanced image dataset and input the imbalanced image dataset into a preset target detector for target detection analysis to obtain unlabeled image classification loss, labeled image classification loss, and pseudo-labels for each unlabeled image; the imbalanced image dataset includes a labeled image subset and an unlabeled image subset; the pseudo-label includes label category, category logical value, and detection box position coordinates;

[0158] The gradient balancing modeling module 2 is used to calculate the partial derivative of the category image classification loss of each label category with respect to the logistic value of each category, obtain the corresponding gradient matrix, and establish the corresponding category gradient balancing equation system based on the gradient matrix; the gradient matrix includes the positive gradient and negative gradient of each label category;

[0159] The classification loss calculation module 3 is used to solve the class gradient balance equation system to obtain the class weights and weight-based loss of each label class, and to obtain the total classification loss based on the class weights and weight-based loss of each label class.

[0160] The resampling training module 4 is used to obtain the corresponding label category threshold according to the category weight of each label category, and to obtain the effective pseudo-label and the image sampling rate of each unlabeled image according to each label category threshold. It is also used to train and update the preset target detector according to the effective pseudo-label, the sampling rate of each image and the total classification loss to obtain a semi-supervised target detector.

[0161] The target detection module 5 is used to acquire the image to be detected and perform target detection on the image to be detected according to the semi-supervised target detector to obtain the detection result.

[0162] Specific limitations regarding the class-imbalanced semi-supervised object detection system can be found in the limitations of the class-imbalanced semi-supervised object detection method described above; the corresponding technical effects are equivalent and will not be repeated here. Each module in the aforementioned class-imbalanced semi-supervised object detection system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0163] Figure 6 An internal structural diagram of a computer device is shown in one embodiment. This computer device may specifically be a terminal or a server. Figure 6As shown, the computer device includes a processor, memory, network interface, display, camera, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a class-imbalanced semi-supervised target detection method. The display screen can be an LCD screen or an e-ink display screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0164] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computing devices may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0165] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0167] In summary, the present invention provides a class-imbalanced semi-supervised object detection method and system. The method acquires a class-imbalanced image dataset comprising labeled and unlabeled image subsets, inputs this dataset into a preset object detector for object detection analysis, and obtains the unlabeled image classification loss, the labeled image classification loss, and the pseudo-labels for each unlabeled image. It then calculates the partial derivatives of the class image classification loss for each labeled class with respect to the logistic values ​​of each class, obtaining gradient matrices including the positive and negative gradients of each labeled class. Based on these gradient matrices, it establishes a corresponding class gradient balance equation system, solves this system to obtain the class weights and weight-based loss for each labeled class, and finally obtains the total classification loss based on the class weights and weight-based loss for each labeled class. After loss, the corresponding label category threshold is obtained according to the category weight of each label category. The effective pseudo-labels and the image sampling rate of each unlabeled image are obtained according to the label category thresholds. The target detector is then resampled, trained and updated according to the effective pseudo-labels, the sampling rates of each image, and the total classification loss. The target detector is then used to perform target detection on the image to be detected, and the detection result is obtained. This method combines a gradient-based reweighting mechanism that balances the positive and negative gradients of the class by estimating the class weights from the gradient matrix with a gradient-based dynamic adaptive threshold strategy and a class-balanced resampling strategy. This effectively solves the two-class confirmation error in class-imbalanced scenarios. It uses unlabeled data to perform reliable target detection for the majority and minority classes, ensuring the accuracy of target detection in class-imbalanced scenarios and improving the application performance of semi-supervised target detection.

[0168] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0169] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A class-imbalanced semi-supervised target detection method, characterized in that, The method includes the following steps: A class-imbalanced image dataset is acquired and input into a preset target detector for target detection analysis to obtain unlabeled image classification loss, labeled image classification loss, and pseudo-labels for each unlabeled image. The class-imbalanced image dataset includes a labeled image subset and an unlabeled image subset. The pseudo-labels include label category, category logical value, and detection box position coordinates. Calculate the partial derivatives of the category image classification loss for each label category with respect to the logistic values ​​of each category to obtain the corresponding gradient matrix, and establish the corresponding category gradient balance equation system based on the gradient matrix; the category image classification loss includes the unlabeled image classification loss and the labeled image classification loss for the corresponding label category; the gradient matrix includes the positive gradient and negative gradient of each label category; Solve the class gradient balance equations to obtain the class weights and weight loss for each label class, and obtain the total classification loss based on the class weights and weight loss for each label class. Based on the category weights of each label category, the corresponding label category thresholds are obtained. Based on the label category thresholds, the effective pseudo-labels and the image sampling rates of each unlabeled image are obtained. Based on the effective pseudo-labels, the image sampling rates, and the total classification loss, the preset target detector is trained and updated to obtain a semi-supervised target detector. The image to be detected is acquired, and the image is subjected to target detection based on the semi-supervised target detector to obtain the detection result.

2. The class-imbalanced semi-supervised target detection method as described in claim 1, characterized in that, The step of calculating the partial derivatives of the category image classification loss for each label category with respect to the logistic values ​​of each category, and obtaining the corresponding gradient matrix, includes: Based on the unlabeled image classification loss and the labeled image classification loss for each label category, the positive gradient and negative gradient for each label category are calculated respectively; the positive gradient is expressed as: in, Indicates label category The positive gradient; Indicates the first The logical value of the category of each candidate box; Indicates the first Pseudo-labels for candidate boxes; Represents category logical value The Middle Tag categories The corresponding logical value score; and These represent the loss function for classified images with and without labels, respectively. The negative gradient is represented as: in, Indicates the label category in the same batch For tag categories The negative gradient, and ; Based on the positive and negative gradients of each label category, the corresponding batch learning gradient matrix is ​​obtained; The gradient matrix is ​​obtained by performing a moving average calculation on the batch learning gradient matrix; the gradient matrix is ​​represented as follows: in, Represents the batch learning gradient matrix; Represents the gradient matrix; This represents the moving average coefficient of the gradient matrix.

3. The class-imbalanced semi-supervised target detection method as described in claim 2, characterized in that, The step of establishing the corresponding class gradient balance equation system based on the gradient matrix includes: Based on the principle that the positive and negative gradients of each label category are equal, the gradient balance equations for each label category are obtained according to the gradient matrix; the gradient balance equations are expressed as follows: in, and They represent the gradient matrices respectively. Chinese label category Positive gradient and label category For tag categories The negative gradient; and Representing the tag categories and tag categories Corresponding category weights; Based on the gradient balance equations for all label categories and the category weight constraint equations, the set of category gradient balance equations is obtained; the category weight constraint equations are expressed as follows: in, Indicates the first Tag categories The corresponding category weights; n represents the total number of categories.

4. The class-imbalanced semi-supervised target detection method as described in claim 3, characterized in that, The steps of solving the class gradient balance equations to obtain the class weights and weight loss for each label class include: Based on the preset learnable category logic, a weight estimation function is generated; the weight estimation function is expressed as: in, This represents the category weight corresponding to the i-th label category; n represents the total number of categories; Indicates and The corresponding learnable category logic; Indicates and The corresponding number One learnable category logic; Based on the weight estimation function, the Jacobian iteration process is used to iteratively solve the class gradient balance equations to obtain the learnable class logic corresponding to each class weight; the Jacobian iteration process is expressed as follows: in, and They represent the first The category weight and corresponding target weight of the i-th label category in each iteration; Based on the learnable category logic corresponding to each category weight and the weight estimation function, the corresponding category weights and weight calculation loss are obtained; the weight calculation loss is expressed as: in, This indicates the weighted loss value.

5. The class-imbalanced semi-supervised target detection method as described in claim 4, characterized in that, The step of calculating the total classification loss based on the category weights of each label category and the weights includes: The class weights of each label category corresponding to the labeled image subset are smoothed to obtain the corresponding labeled class weights; the labeled class weights are expressed as follows: in, and These represent the class weight of the i-th labeled category and the corresponding labeled class weight, respectively; Indicates hyperparameters; The image classification loss is obtained based on the weights of each labeled category and the weights of each label category corresponding to the labeled image subset. The total classification loss is obtained by summing the image classification loss and the weight solving loss; the total classification loss is expressed as: In the formula, in, , and These represent the total classification loss, the image classification loss, and the weight calculation loss, respectively. This indicates that there are labeled category weights.

6. The class-imbalanced semi-supervised target detection method as described in claim 1, characterized in that, The step of obtaining the corresponding tag category threshold based on the category weight of each tag category includes: Based on a gradient adaptive strategy, the corresponding label category threshold is obtained according to the category weight of each label category; the label category threshold is expressed as: in, This is a preset threshold hyperparameter; This represents the label category threshold for the i-th label category; This represents the category weight of the i-th label category.

7. The class-imbalanced semi-supervised target detection method as described in claim 1, characterized in that, The step of obtaining the effective pseudo-labels and the image sampling rates of each unlabeled image based on the threshold of each label category includes: The pseudo-labels of each unlabeled image are filtered according to the label category threshold to obtain the effective pseudo-labels; Based on the category logical value of each valid pseudo-label and the corresponding label category threshold, the corresponding unlabeled category resampling rate is obtained; the unlabeled category resampling rate is expressed as: In the formula, in, Indicates label category The corresponding unlabeled category resampling rate; and These represent the tag category and category logical value of a valid pseudo-tag, respectively; Indicates label category The label category threshold; A coefficient indicating the range of the adjusted sampling rate; Indicates the total number of training iterations; Indicates the current training iteration round number; This indicates the preset dynamic sampling threshold; and These represent the number of labeled images and the number of unlabeled images in the class-imbalanced image dataset, respectively. This indicates that the class-imbalanced image dataset contains at least one labeled class. Number of images with bounding boxes; The maximum value of the unlabeled category resampling rate for each unlabeled image is selected as the corresponding image sampling rate.

8. A class-imbalanced semi-supervised target detection system, characterized in that, The system includes: The pseudo-label generation module is used to acquire an imbalanced image dataset and input the imbalanced image dataset into a preset target detector for target detection analysis to obtain unlabeled image classification loss, labeled image classification loss, and pseudo-labels for each unlabeled image. The imbalanced image dataset includes a labeled image subset and an unlabeled image subset. The pseudo-labels include label category, category logical value, and detection box position coordinates. The gradient balance modeling module is used to calculate the partial derivatives of the category image classification loss for each label category with respect to the logistic values ​​of each category, obtain the corresponding gradient matrix, and establish the corresponding category gradient balance equation system based on the gradient matrix; the category image classification loss includes the unlabeled image classification loss and the labeled image classification loss for the corresponding label category; the gradient matrix includes the positive gradient and negative gradient of each label category; The classification loss calculation module is used to solve the class gradient balance equation system to obtain the class weights and weight-based loss of each label class, and to obtain the total classification loss based on the class weights and weight-based loss of each label class. The resampling training module is used to obtain the corresponding label category threshold according to the category weight of each label category, and to obtain the effective pseudo-label and the image sampling rate of each unlabeled image according to each label category threshold. The module is also used to train and update the preset target detector according to the effective pseudo-label, the image sampling rate and the total classification loss to obtain a semi-supervised target detector. The target detection module is used to acquire the image to be detected and perform target detection on the image to be detected according to the semi-supervised target detector to obtain the detection result.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images