Weakly supervised rt-detr object detection method based on pseudo label improvement

Abstract

The application relates to the technical field of real-time target detection and discloses a weakly supervised RT-DETR target detection method based on improved pseudo labels. The method generates pseudo labels by weakly supervising label data sets and enhances the pseudo labels, and an improved model comprising a lightweight backbone network, a pseudo label guided cross-scale interaction module and a decoder with a pseudo label position constraint mechanism is constructed. The model is trained end to end by adopting an iterative pseudo label updating strategy. The pseudo label guided cross-scale interaction module dynamically modulates the feature fusion process by using pseudo label information, thereby enhancing the feature discrimination ability of the model for multi-scale targets. The decoder with the pseudo label position constraint mechanism introduces the geometric prior of the pseudo label as a soft constraint into query optimization, thereby improving the positioning accuracy and training stability of the model under weak supervision and reducing the dependence on accurate labeled data.

Description

A Weakly Supervised RT-DETR Target Detection Method Based on Pseudo-Labels Technical Field

[0001] This invention relates to the field of real-time target detection technology, specifically to a weakly supervised RT-DETR target detection method based on pseudo-label improvement. Background Technology

[0002] Real-time object detection models like RT-DETR have attracted attention due to their end-to-end architecture and high performance; however, their training heavily relies on large amounts of accurately labeled data, which is costly to obtain. Utilizing incomplete or noisy weak labels for learning is a key path to alleviate this dependence. Existing methods often introduce pseudo-label techniques into the RT-DETR framework, using the model's own predictions as supervision signals. However, the standard RT-DETR architecture is not designed for handling noisy weak supervision signals, and directly applying pseudo-labels will face inherent limitations.

[0003] The classification and location noise contained in pseudo-labels accumulates and propagates during training. When the encoder of a standard RT-DETR performs cross-scale feature fusion, the process is entirely data-driven, lacking prior guidance on potential target regions. This leads to the possibility that background noise features may be incorrectly amplified under weak supervision, while the discriminative features of the true target are not effectively aggregated, limiting the model's feature representation ability. Its decoder uses Hungarian matching for one-to-one supervision, allowing for a highly flexible optimization process. When the quality of pseudo-labels is uneven, the optimization of object queries is easily misled by incorrect labels, causing predicted boxes to deviate significantly from the true target, impairing the stability and convergence of the model's localization.

[0004] There is a need for an RT-DETR method specifically designed for weakly supervised scenarios. This method should be able to use weak signals for guidance during the feature fusion stage and impose reasonable constraints on position prediction during the decoding stage, thereby achieving stable training and improved accuracy under conditions of scarce annotations. Summary of the Invention

[0005] The purpose of this invention is to provide a weakly supervised RT-DETR target detection method based on pseudo-label improvement, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a weakly supervised RT-DETR target detection method based on pseudo-label improvements, the method comprising:

[0007] Obtain an open-source dataset containing image data, perform weakly supervised annotation on the open-source dataset to generate initial pseudo-labels, and perform data augmentation operations on the image data and initial pseudo-labels to obtain augmented training image data and corresponding augmented pseudo-labels.

[0008] An improved RT-DETR model architecture is constructed, which includes a lightweight backbone network, a pseudo-label-guided cross-scale interaction module, and a decoder with a pseudo-label position constraint mechanism.

[0009] The enhanced training image data and the corresponding enhanced pseudo-labels are input into the improved RT-DETR model architecture. An iterative pseudo-label update strategy is used to train the improved RT-DETR model architecture end-to-end. During the training process, the computational cost of forward and backward propagation is controlled in real time to obtain the trained improved RT-DETR object detection model.

[0010] The image to be detected is input into the trained improved RT-DETR object detection model to obtain the object detection result.

[0011] Preferably, the open-source dataset is subjected to weakly supervised annotation to generate initial pseudo-labels, and data augmentation operations are performed on the image data and the initial pseudo-labels to obtain augmented training image data and corresponding augmented pseudo-labels, including:

[0012] Image data and corresponding raw annotation information are extracted from the open-source dataset. The raw annotation information includes the target category and bounding box coordinates.

[0013] A pre-trained general object detection model is used to infer the image data and generate preliminary prediction boxes. The preliminary prediction boxes are then matched and filtered with the original annotation information based on the degree of overlap. Prediction boxes with a confidence level higher than a preset threshold are retained as weakly supervised annotation results.

[0014] The weakly supervised annotation results are converted into annotation data with the same format as the original annotation information, and used as initial pseudo-labels;

[0015] Random horizontal flipping, random cropping, and color perturbation operations are applied simultaneously to the image data and the initial pseudo-labels to generate enhanced training image data and corresponding enhanced pseudo-labels.

[0016] Preferably, an improved RT-DETR model architecture is constructed, comprising a lightweight backbone network, a pseudo-label-guided cross-scale interaction module, and a decoder with a pseudo-label position constraint mechanism, including:

[0017] A lightweight backbone network is constructed as a feature extractor to perform multi-scale feature extraction on the input image and output a multi-scale feature map.

[0018] A pseudo-label-guided cross-scale interaction module is connected after the multi-scale feature map output by the lightweight backbone network. The pseudo-label-guided cross-scale interaction module uses the input multi-scale feature map and the enhanced pseudo-label used in the current training iteration to calculate the cross-scale feature fusion weights and generate the fused multi-scale interactive feature map.

[0019] A decoder with a pseudo-label position constraint mechanism is constructed. The decoder receives the fused multi-scale interactive feature map and incorporates the spatial position information of the enhanced pseudo-label during its attention calculation process to generate the decoded target query features.

[0020] Preferably, the lightweight backbone network includes multiple convolutional stages, and the feature map output by each convolutional stage serves as a feature map at different scales, thus forming the multi-scale feature map.

[0021] The pseudo-label-guided cross-scale interaction module utilizes the input multi-scale feature map and the enhanced pseudo-labels used in the current training iteration to calculate cross-scale feature fusion weights and generate a fused multi-scale interaction feature map, including:

[0022] Based on the size and position information of the target bounding box in the enhanced pseudo-label, calculate the probability distribution of the target's presence on feature maps at different scales;

[0023] Based on the existence probability distribution, a guiding mask is generated for each spatial location of each scale feature map;

[0024] The guiding mask is used to weight the corresponding multi-scale feature maps, and the fused multi-scale interactive feature map is generated by cross-scale feature addition and convolution operations.

[0025] Preferably, the decoder with the pseudo-label position constraint mechanism incorporates the spatial position information of the enhanced pseudo-labels during its attention calculation process to generate decoded target query features, including:

[0026] In each decoding layer of the decoder, cross-attention between the target query features and the fused multi-scale interaction feature map is calculated;

[0027] Based on the center point coordinates of all target instances in the enhanced pseudo-label, a spatial position offset matrix is ​​generated;

[0028] The spatial position bias matrix and the cross attention weight matrix are added element by element to obtain the position-constrained attention weight matrix;

[0029] The attention weight matrix after the position constraint is used to perform weighted summation on the fused multi-scale interaction feature map to update the target query feature. After iterative updates through multiple decoding layers, the final decoded target query feature is output.

[0030] Preferably, an iterative pseudo-label update strategy is used to train the improved RT-DETR model architecture end-to-end, and the computational cost of forward and backward propagation is controlled in real time during training, including:

[0031] The enhanced training image data is divided into multiple training batches;

[0032] In the first training iteration, the enhanced pseudo-labels are used as supervision signals to calculate the model prediction loss and perform backpropagation to update the model parameters;

[0033] At the beginning of each subsequent training iteration, the improved RT-DETR model architecture under the current parameter state is used to perform forward inference on the enhanced training image data to generate prediction results;

[0034] Non-maximum suppression is applied to the prediction results, and high-quality prediction boxes are selected based on the prediction confidence and the overlap with the current enhanced pseudo-labels. The high-quality prediction boxes are then used to update the corresponding part of the enhanced pseudo-labels of the image, generating the updated enhanced pseudo-labels.

[0035] Using the updated enhanced pseudo-labels as new supervision signals, the model prediction loss is calculated, and subsequent training iterations are performed.

[0036] Preferably, the computational complexity of forward and backward propagation is controlled in real time during training, including:

[0037] During the forward inference process of each training iteration, the intermediate feature maps in the improved RT-DETR model architecture are dynamically sampled at different resolutions, and the spatial resolution of the sampling is adjusted according to the preset computational budget.

[0038] During backpropagation, only the gradients of some layers in the improved RT-DETR model architecture are calculated and backpropagated. These layers are dynamically selected based on their parameter updates to determine their contribution to the decrease of the loss function.

[0039] Preferably, the image to be detected is input into the trained improved RT-DETR object detection model to obtain object detection results, including:

[0040] The image to be detected is normalized to generate a standard-sized input image;

[0041] The standard-sized input image is input into the trained improved RT-DETR target detection model. The lightweight backbone network extracts multi-scale feature maps, and the pseudo-label-guided cross-scale interaction module generates a fused multi-scale interactive feature map. Finally, the decoder with pseudo-label position constraint mechanism outputs the final target query features.

[0042] The final target query features are input into a prediction head, which outputs the target category probability and bounding box coordinates as preliminary target detection results.

[0043] A non-maximum suppression operation is performed on the preliminary target detection results to eliminate redundant detection boxes and obtain the final target detection results.

[0044] Preferably, the prediction head consists of two fully connected layer branches, one fully connected layer branch is used to predict the target class probability, and the other fully connected layer branch is used to predict the bounding box coordinates;

[0045] The target class probability output by the prediction head is normalized by the softmax function, and the bounding box coordinates are normalized by the sigmoid function.

[0046] Preferably, the method further includes:

[0047] After obtaining the trained improved RT-DETR object detection model, the performance of the trained improved RT-DETR object detection model is evaluated using an additional validation dataset;

[0048] The evaluation process includes calculating the average accuracy metric of the trained improved RT-DETR object detection model on the validation dataset and comparing it with the average accuracy metric of the unimproved original RT-DETR model on the same validation dataset.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] The pseudo-label-guided cross-scale interaction module embeds pseudo-labels generated under weak supervision as attention modulation signals into the feature fusion process of the efficient hybrid encoder. This technique enables the model to dynamically adjust the contribution weights of different feature maps when integrating high-level semantic features and low-level detail features, focusing on the foreground region indicated by the pseudo-labels. The cross-scale fusion in the feature pyramid network thus becomes goal-oriented, enhancing the representation ability of multi-scale targets, especially exhibiting stronger robustness in scenarios with small and blurred targets. The feature representation output by the encoder contains richer contextual information, improving the quality of the initial query selection.

[0051] The decoder with a pseudo-label position constraint mechanism introduces geometric consistency regularization during query optimization. This mechanism uses the coarse bounding box information provided by the pseudo-labels as a soft constraint for the decoder's position query optimization, penalizing severe deviations between the predicted bounding box and the prior position of the pseudo-label. When updating the target query, the decoder not only needs to minimize the detection loss but also maintain the consistency between the predicted result and the pseudo-label in spatial distribution. This constraint stabilizes the training process, reduces the divergence of optimization direction caused by sparse weak supervision signals, and improves the model's localization accuracy in complex scenes. The iterative optimization of object queries is more stable, and the final predicted bounding boxes have higher recall and localization accuracy. Attached Figure Description

[0052] Figure 1 is a schematic diagram illustrating the working principle of the weakly supervised RT-DETR target detection method based on pseudo-label improvement described in this invention.

[0053] Figure 2 is a flowchart of the initial pseudo-label generation and data augmentation process;

[0054] Figure 3 is a flowchart of constructing the improved RT-DETR model architecture;

[0055] Figure 4 shows a comparison of the number of channels in the multi-scale feature maps of the RT-DETR target detection model.

[0056] Figure 5 shows a comparison of the computational efficiency of different target detection models. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Referring to Figure 1, this invention provides a weakly supervised RT-DETR object detection method based on improved pseudo-labels. The method includes: acquiring an open-source dataset containing image data; performing weakly supervised annotation processing on the open-source dataset to generate initial pseudo-labels; and performing data augmentation operations on the image data and the initial pseudo-labels to obtain augmented training image data and corresponding augmented pseudo-labels. An improved RT-DETR model architecture is constructed, integrating a lightweight backbone network, a pseudo-label-guided cross-scale interaction module, and a decoder with a pseudo-label position constraint mechanism. The augmented training image data and corresponding augmented pseudo-labels are input into the improved RT-DETR model architecture, and end-to-end training is performed using an iterative pseudo-label update strategy. During training, the computational cost of forward and backward propagation is controlled in real time, ultimately obtaining a trained improved RT-DETR object detection model. The object detection result is obtained by inputting the image to be detected into the trained improved RT-DETR object detection model.

[0059] In one embodiment of the present invention, referring to Figure 2, 100,000 image data and corresponding original annotation information are extracted from the publicly available MSCOCO dataset. The original annotation information includes the target category and bounding box coordinates. The bounding box coordinates are stored in the format of top-left x-coordinate, top-left y-coordinate, width, and height. A Faster R-CNN model pre-trained on the ImageNet dataset is used to infer the 100,000 image data, generating preliminary predicted boxes for each image. Each preliminary predicted box includes bounding box coordinates, predicted category, and confidence score. The preliminary predicted boxes are matched and filtered with the original annotation information based on overlap. The overlap is calculated using the intersection-union ratio (IU), which is defined as the ratio of the intersection area of ​​the predicted box and the original annotation box to the union area. Predicted boxes with a confidence score higher than a preset threshold of 0.7 are retained as weakly supervised annotation results. For a predicted box that matches multiple original annotation boxes in an image, the category of the original annotation box with the highest IU is selected as the category label of the predicted box. The weakly supervised annotation results are converted into annotation data with the same format as the original annotation information, i.e., a text file containing the target category and bounding box coordinates, as the initial pseudo-labels.

[0060] In some embodiments, a random horizontal flip operation is simultaneously applied to the image data and the initial pseudo-label. The probability of the random horizontal flip is set to 0.5. When the random number is less than 0.5, the image data is flipped along the vertical central axis, and the horizontal coordinate of the bounding box in the initial pseudo-label is mirrored synchronously. A random cropping operation is simultaneously applied to the image data and the initial pseudo-label. The area ratio of the cropping region is set between 0.64 and 1.0 of the original image area, and the aspect ratio of the cropping region is set between 0.75 and 1.33. A rectangular region that meets the area and aspect ratio requirements is randomly sampled from the image. The image data is cropped from this rectangular region. Bounding boxes in the initial pseudo-label that are completely outside the rectangular region are removed, and bounding boxes that are partially within the rectangular region are cropped to the boundary of the rectangular region. Color perturbation operations are applied simultaneously to the image data and the initial pseudo-label. The color perturbation operations include adjusting the image brightness, contrast, and saturation. The brightness adjustment factor is uniformly sampled between 0.8 and 1.2, the contrast adjustment factor is uniformly sampled between 0.8 and 1.2, and the saturation adjustment factor is uniformly sampled between 0.8 and 1.2. Each pixel value of the image data is linearly transformed according to the three factors, and the value of the initial pseudo-label remains unchanged during this process.

[0061] It is understandable that the matching and filtering process between the weakly supervised annotation results and the original annotation information relies on the Cross-Union Ratio (CUI) threshold, which is set to 0.5. When the CUI of the predicted bounding box and the original labeled bounding box is greater than or equal to 0.5, the predicted bounding box is considered to match the original labeled bounding box. For successfully matched predicted bounding boxes, their category labels inherit the category labels of the original labeled bounding boxes, and their bounding box coordinates are included as part of the weakly supervised annotation result. The matching and filtering process employs a bidirectional optimal matching strategy: for each original labeled bounding box, among all predicted bounding boxes with an CUI greater than 0.5, the predicted bounding box with the highest confidence is selected for matching; for each predicted bounding box, among all original labeled bounding boxes with an CUI greater than 0.5, the original labeled bounding box with the highest CUI is selected for matching. In the final weakly supervised annotation result, each target instance corresponds to one filtered predicted bounding box.

[0062] In some embodiments, a preset threshold is used to filter the confidence of predicted bounding boxes, and the preset threshold is set to 0.7. Predicted bounding boxes with a confidence level higher than 0.7 are retained, while those with a confidence level lower than or equal to 0.7 are discarded. The retained predicted bounding boxes constitute the weakly supervised annotation results, which are used to generate initial pseudo-labels. For a single image, a pre-trained general object detection model may generate hundreds of initial predicted bounding boxes. After dual filtering based on the confidence threshold and the intersection-union ratio (IU / U) threshold, the final number of retained predicted bounding boxes is typically between ten and several dozen. The data format of the initial pseudo-labels is completely consistent with the original annotation format of the MSCOCO dataset, including the image file name, image size information, and the class identifier and bounding box coordinates for each object instance.

[0063] Optionally, the random cropping operation in data augmentation is applied to both the image data and the initial pseudo-labels. During the random cropping operation, the bounding box coordinates in the initial pseudo-labels are adjusted synchronously according to the coordinate offset of the cropped region. The bounding box coordinates are adjusted by subtracting the x-coordinate of the cropped region from the x-coordinate of the top-left corner of the bounding box, and subtracting the y-coordinate of the cropped region from the y-coordinate of the top-left corner of the bounding box, while keeping the width and height of the bounding box unchanged. If a portion of the adjusted bounding box extends beyond the boundary of the cropped image, the coordinates of the excess portion are truncated to the image boundary. After the random cropping operation, the bounding box coordinates in the augmented pseudo-labels always remain consistent with the target spatial location in the augmented training image data.

[0064] Optionally, the color perturbation operation is applied only to image data, and does not change the initial pseudo-label values. After the color perturbation operation, the pixel values ​​of the image data change, but the initial pseudo-labels remain unchanged. The color perturbation operation is performed in the RGB color space of the image, independently applying adjustments to the brightness, contrast, and saturation factors for each color channel. Brightness adjustment is achieved by multiplying the RGB value of each pixel by the same brightness factor; contrast adjustment is achieved by stretching or compressing pixel values ​​towards grayscale values ​​through linear interpolation; and saturation adjustment is achieved by stretching or compressing pixel values ​​towards grayscale values ​​with the same brightness through linear interpolation. The enhanced training image data thus gains color diversity, while the coordinates and category information of the corresponding enhanced pseudo-labels remain unaffected by the color changes.

[0065] It is understandable that the generation of initial pseudo-labels is the starting point of an iterative optimization process, and the quality of the initial pseudo-labels affects the subsequent model training effect. The initial pseudo-labels contain target category and bounding box coordinate information. The target category information comes from the original annotation information, and the bounding box coordinate information comes from the prediction results of a pre-trained general object detection model. In specific implementation, preliminary prediction boxes are generated through the pre-trained model. The original category information is associated with the prediction box coordinates through overlap-based matching filtering. Low-quality predictions are filtered out through a confidence threshold, ultimately forming well-formatted initial pseudo-labels. The same geometric transformation enhancement operation is applied to the image data and the initial pseudo-labels to ensure that the enhanced pseudo-labels are spatially aligned with the enhanced image. Applying color perturbation only changes the appearance of the image without affecting the label, thereby obtaining the enhanced training image data and corresponding enhanced pseudo-labels for training.

[0066] In one embodiment of the present invention, referring to Figure 3, constructing the improved RT-DETR model architecture includes constructing a lightweight backbone network as a feature extractor. The lightweight backbone network adopts the MobileNetV3-Small structure and contains five convolutional stages. The input image size is 640 pixels by 640 pixels. The first convolutional stage outputs a feature map with a size of 320 pixels by 320 pixels and 16 channels. The second convolutional stage outputs a feature map with a size of 160 pixels by 160 pixels and 24 channels. The third convolutional stage outputs a feature map with a size of 80 pixels by 80 pixels and 48 channels. The fourth convolutional stage outputs a feature map with a size of 40 pixels by 40 pixels and 96 channels. The fifth convolutional stage outputs a feature map with a size of 20 pixels by 20 pixels and 96 channels. These five feature maps constitute a multi-scale feature map for subsequent processing. In practical implementation, the pseudo-label-guided cross-scale interaction module utilizes the input multi-scale feature maps and the enhanced pseudo-labels used in the current training iteration to calculate cross-scale feature fusion weights. The enhanced pseudo-labels contain the coordinate information of the target bounding box. For a target bounding box in the input image, the area of ​​the target bounding box is calculated, and the matching degree between the target bounding box and the preset anchor size of each scale feature map is calculated. The preset anchor size corresponds to the size of the image region covered by the receptive field of different scale feature maps. For example, for a feature map with a size of 80 pixels by 80 pixels, its preset anchor size is 256 pixels by 256 pixels. The probability distribution of the existence of the target on different scale feature maps is calculated through a function based on the ratio of the target bounding box size to the preset anchor size. The specific calculation relationship is expressed by the following formula:

[0067]

[0068] in: This represents the probability of the target's existence on the feature map s at the current scale. This represents the area of ​​the target bounding box. This represents the preset anchored area corresponding to the scale feature map s. This is a hyperparameter that controls the sharpness of the probability distribution, and in practice, it is set to 5.0. Based on the calculated existence probability distribution... A guiding mask is generated for each spatial location in the feature map at each scale. The guiding mask is generated by initializing all locations on the feature map to 0. For each target bounding box in the enhanced pseudo-label, a guiding mask is generated around its location on the corresponding scale feature map based on its center point coordinates projected onto that location, using a mask with a probability of existence. Within the determined square area, the value of the boot mask will be set to... The corresponding multi-scale feature maps are weighted using a guiding mask. The weighting method is to multiply each channel of the feature map by the guiding mask value of the corresponding spatial location. Then, the weighted feature maps of all scales are upsampled to the same spatial size and concatenated along the channel dimension. Finally, they are fused through a convolutional layer with a kernel size of 3 by 3 to generate a fused multi-scale interactive feature map with 256 channels.

[0069] In some embodiments, the decoder with a pseudo-label position constraint mechanism receives the fused multi-scale interactive feature map and incorporates the spatial position information of the enhanced pseudo-labels during its attention calculation. The decoder contains six identical decoding layers, each containing a self-attention sublayer and a cross-attention sublayer. During the calculation of the cross-attention sublayer, the cross-attention between the target query features and the fused multi-scale interactive feature map is calculated. The target query features are a set of learnable vectors, with a quantity of 100 and a dimension of 256. The calculation of the cross-attention weights first involves performing a dot product operation between the linear transformation matrix of the target query features and the linear transformation matrix of the fused multi-scale interactive feature map to obtain an initial attention weight matrix. A spatial location bias matrix is ​​generated based on the center point coordinates of all target instances in the enhanced pseudo-labels. The generation process involves creating a zero matrix with the same spatial size as the fused multi-scale interactive feature map. For each target instance in the enhanced pseudo-labels, its center point coordinates are mapped to a grid position in the feature map according to the feature map downsampling rate. Within a circular region centered at this grid position with a radius of 3 grid units, a negative constant bias value of -10 is assigned to the corresponding position in the location bias matrix. The location bias value outside the circular region is set to 0. The spatial location bias matrix is ​​element-wise added to the initial cross-attention weight matrix to obtain the location-constrained attention weight matrix. The location-constrained attention weight matrix is ​​then normalized using the Softmax function. The normalized location-constrained attention weight matrix is ​​used to perform a weighted summation on the fused multi-scale interactive feature map to update the target query features. After iterative updates through six decoding layers, the final decoded target query features are output.

[0070] In some embodiments, each convolutional stage of the lightweight backbone network consists of multiple improved inverse residual blocks, which include depthwise separable convolutions, squeezed excitation attention modules, and skip connections. The total number of parameters in the lightweight backbone network is kept below three million to ensure computational efficiency in the feature extraction process. Multi-scale feature maps are output from the ends of the second, third, fourth, and fifth convolutional stages of the lightweight backbone network, corresponding to feature maps with downsampling rates of 8, 16, 32, and 64, respectively. These multi-scale feature maps are fed into a pseudo-label-guided cross-scale interaction module for further processing.

[0071] It is understandable that the guidance mask generation process in the pseudo-label-guided cross-scale interaction module relies on real-time information from enhanced pseudo-labels. In each training iteration, the enhanced pseudo-labels of the input image are converted into spatial guidance masks corresponding to feature maps at each scale. The value of the guidance mask reflects the prior probability of the target existing on feature maps at different scales; a higher value indicates a greater probability that the corresponding location in the feature map at that scale contains target information. During the weighting process, the feature map is enhanced at locations with high guidance mask values ​​and suppressed at locations with low guidance mask values. This achieves the goal of using weakly supervised pseudo-label information to guide the fusion direction of multi-scale features, making the fused multi-scale interactive feature map more focused on regions that may contain the target.

[0072] It is understandable that the spatial bias matrix in a decoder with a pseudo-label position constraint mechanism serves to provide spatial priors for the attention mechanism. The spatial bias matrix applies a negative bias constant to the target center region indicated by the enhanced pseudo-label. This bias is added to the initial attention weights before Softmax normalization, significantly reducing the attention weights for locations outside the target center region. This guides the decoder's attention to focus more on the vicinity of the potential target location labeled by the enhanced pseudo-label. This mechanism integrates the weakly supervised pseudo-label position information as a soft constraint into the attention calculation process of the Transformer decoder, enabling the target query features to more effectively focus on relevant image regions during updates.

[0073] Optionally, when generating the guidance mask, the pseudo-label-guided cross-scale interaction module may have a probability distribution for targets with bounding box areas less than ten pixels in the enhanced pseudo-label. The value is forced to be 0, meaning the target does not participate in the guided mask calculation of any scale feature map. This process avoids unreliable existence probability estimates from excessively small target bounding boxes, which may only correspond to one or two pixels on the feature map, resulting in significant noise in their positional information. After the guided mask weighting operation, feature maps of different scales are upsampled to the same size of 80 pixels by 80 pixels through bilinear interpolation for channel-dimensional stitching and subsequent convolutional fusion operations.

[0074] Optionally, the negative constant bias value can be adjusted when generating the spatial location bias matrix in the decoder with pseudo-label position constraint mechanism. The larger the absolute value of the negative constant bias value, the stronger the attention suppression on non-target regions. In a specific implementation, the negative constant bias value is set to -10. This value makes the attention weights at locations outside the target region sufficiently small before Softmax, thus bringing them close to zero after normalization. The spatial location bias matrix is ​​only used during the model training phase. When using the trained improved RT-DETR object detection model for inference, the decoder's cross-attention calculation does not incorporate the spatial location bias matrix; only the initial cross-attention weight matrix is ​​used for calculation and normalization.

[0075] In one embodiment of the present invention, an iterative pseudo-label update strategy is employed to train the improved RT-DETR model architecture end-to-end. The enhanced training image data is divided into multiple training batches, each containing sixteen images. In the first training iteration, the enhanced pseudo-labels are used as supervision signals to calculate the model prediction loss, which includes classification loss and bounding box regression loss. Backpropagation is then performed to update the parameters of the improved RT-DETR model architecture. At the beginning of each subsequent training iteration, the improved RT-DETR model architecture with the current parameter state is used to perform forward inference on the enhanced training image data to generate prediction results. The prediction results include target class probabilities and bounding box coordinates. Non-maximum suppression (NMS) processing is applied to the prediction results, with an intersection-overlap ratio (IoU) threshold of 0.5. High-quality prediction boxes are selected based on prediction confidence and overlap with the current enhanced pseudo-labels, with a prediction confidence threshold of 0.8. The overlap with the current enhanced pseudo-labels is calculated using the IoU, with an IoU threshold of 0.6. High-quality predicted bounding boxes are used to update the corresponding image's partially enhanced pseudo-labels, generating updated enhanced pseudo-labels. The update method involves replacing the original enhanced pseudo-labels with high-quality predicted bounding boxes that correspond to the original enhanced pseudo-labels with an intersection-union ratio (IU) higher than 0.7. The updated enhanced pseudo-labels are used as new supervision signals to calculate the model prediction loss and execute subsequent training iterations. In practice, the computational cost of forward and backward propagation is controlled in real-time during training. During the forward inference process of each training iteration, dynamic resolution sampling is performed on the intermediate feature maps in the improved RT-DETR model architecture. These intermediate feature maps come from the output of the lightweight backbone network. Dynamic resolution sampling adjusts the spatial resolution of the sampling according to a preset computational budget, measured in floating-point operations (FOPs), set to one billion FOPs per second. The dynamic resolution sampling process calculates the target sampling resolution using a formula, expressed as:

[0076]

[0077] in: Indicates the first The spatial resolution of the feature map sampled in each training iteration, i.e., the number of pixels in the height and width of the feature map. This represents the spatial resolution of the original feature map. This represents the budget for real-time computation available in the current training iteration. This represents the baseline computational workload budget, set at two billion floating-point operations per second. Based on the calculated... Bilinear interpolation is used to adjust the spatial size of the intermediate feature map to... Multiply The pixels are then processed before subsequent forward computation. During backpropagation, gradients are calculated and backpropagated only for a subset of layers in the improved RT-DETR model architecture. These layers are dynamically selected based on their contribution to the loss function descent according to their parameter updates. The contribution is estimated by averaging the gradient norms of each layer's parameters over the last five training iterations. The gradient norm is the L2 norm of all weight gradient vectors of that layer. Before backpropagation begins in each training iteration, the top ten layers with the highest contributions are selected for complete gradient calculation and backpropagation. For unselected layers, their gradients are set to zero and their parameters are not updated.

[0078] In some embodiments, the training batch size can be adjusted according to hardware memory, and the training batch size is set to eight or sixteen images. The total number of training rounds for the iterative pseudo-label update strategy is set to fifty rounds, with enhanced pseudo-labels updated after each training iteration. During the high-quality prediction box selection process, the prediction confidence threshold and overlap threshold can be adjusted, with the prediction confidence threshold between 0.7 and 0.9, and the overlap threshold between 0.5 and 0.7. The cross-union threshold for non-maximum suppression processing is fixed at 0.5. In some embodiments, the baseline computational cost of dynamic resolution sampling is budgeted. The baseline computational cost can be adjusted based on the performance of the training hardware. Set to perform five billion floating-point operations per second to allow for higher sampling resolution. The square root operation in the formula ensures a sublinear relationship between resolution adjustment and computational load changes, when real-time computational load is budgeted. Budget below the baseline calculation volume At this time, the sampling resolution is reduced proportionally. The number of iterations for contribution estimation in selective gradient backpropagation can vary, and the contribution estimation uses the average gradient norm of the most recent ten training iterations.

[0079] It's understandable that the iterative pseudo-label update strategy improves the quality of pseudo-labels in weakly supervised scenarios by gradually optimizing the training labels using the model's own predictions. The updated enhanced pseudo-labels incorporate the knowledge learned during training, ensuring that the supervision signal optimizes synchronously as the model's capabilities improve. The real-time control of forward and backward propagation computation mechanisms allows the training process to adapt to different computational environments. When computational resources are limited, dynamic resolution sampling reduces the feature map space size, decreasing the forward computation burden, while selective gradient backpropagation reduces the computational cost of backpropagation. These operations collectively ensure that the training process continues within a limited computational budget, while gradually improving model performance through the iterative pseudo-label update strategy.

[0080] Optionally, in high-quality predicted bounding box selection, a stricter confidence threshold of 0.85 and overlap threshold of 0.65 can be used to generate more reliable updated augmented pseudo-labels. The updated augmented pseudo-labels only replace the portions of the original augmented pseudo-labels with confidence levels lower than the predicted bounding boxes, retaining the high-confidence labeled boxes from the original augmented pseudo-labels. Optionally, in dynamic resolution sampling, the spatial resolution of the original feature map... It can take feature maps of different scales from the output of the lightweight backbone network. For example, for an 80-pixel by 80-pixel feature map, the spatial resolution of the original feature map is... Set to 80. Real-time computational budget. The utilization rate of the training hardware is dynamically obtained and updated every five seconds.

[0081] In one embodiment of the present invention, the image to be detected is normalized to generate a standard-sized input image. The original size of the image to be detected may be 1920 pixels by 1080 pixels. The size normalization process scales the image to a fixed size of 640 pixels by 640 pixels using bilinear interpolation to generate the standard-sized input image. The standard-sized input image is then input into the trained improved RT-DETR object detection model. The standard-sized input image is processed by a lightweight backbone network to extract multi-scale feature maps. The lightweight backbone network outputs feature maps at four scales, with sizes of 80 pixels by 80 pixels, 40 pixels by 40 pixels, 20 pixels by 20 pixels, and 10 pixels by 10 pixels. The multi-scale feature maps are then processed by a pseudo-label-guided cross-scale interaction module to generate a fused multi-scale interactive feature map. The spatial size of the fused multi-scale interactive feature map is 80 pixels by 80 pixels, and the number of channels is 256. The fused multi-scale interactive feature map is finally output by a decoder with a pseudo-label position constraint mechanism, resulting in the final target query feature, which is a set of 100 256-dimensional vectors. This final target query feature is input into a prediction head, which consists of two fully connected layer branches. The first fully connected layer branch predicts the target class probability; its input dimension is 256, and its output dimension is the total number of object classes plus 1. The second fully connected layer branch predicts the bounding box coordinates; its input dimension is 256, and its output dimension is 4. The target class probability output by the prediction head is normalized using a softmax function, and the bounding box coordinates are normalized using a sigmoid function. The sigmoid function maps the bounding box coordinate values ​​to a range of 0 to 1, representing the relative position and size with respect to the input image size of 640 pixels multiplied by 640 pixels. The result output by the prediction head serves as the preliminary target detection result, containing 100 detection boxes, each accompanied by a class probability vector and a normalized bounding box coordinate vector.

[0082] In some embodiments, non-maximum suppression (NMS) is performed on the preliminary object detection results to eliminate redundant detection boxes, obtaining the final object detection result. The intersection-union (IU) threshold for NMS is set to 0.5, and NMS is performed independently for each object category. The NMS process is as follows: for each category, all detection boxes are sorted from high to low probability, the detection box with the highest category probability is selected as the benchmark, the IU of this detection box with all remaining detection boxes is calculated, and detection boxes with an IU greater than 0.5 are removed. The above benchmark selection and removal process is repeated for the remaining detection boxes until all detection boxes have been processed. After NMS, the number of detection boxes retained for each category is variable, and the detection boxes retained for all categories constitute the final object detection result. The final object detection result is output in list form, where each element contains the category label, confidence score, and bounding box coordinates in absolute pixel coordinates. The bounding box coordinates are obtained by multiplying the normalized coordinates by 640. Refer to Table 1, which shows a final object detection result containing three detected objects.

[0083] Table 1: Final Target Detection Results

[0084]

[0085] In some embodiments, the target class probabilities output by the prediction head are normalized using a softmax function. The softmax function calculates the class probability vector for each detection box, ensuring that the sum of the probabilities for all classes is 1. The bounding box coordinates output by the prediction head are normalized using a sigmoid function, which maps any real number output by the fully connected layer to a value between 0 and 1. The four values ​​of the bounding box coordinates represent the ratio of the center point's x-coordinate to the image width, the ratio of the center point's y-coordinate to the image height, the ratio of width to image width, and the ratio of height to image height, respectively. A standard-size input image is 640 pixels by 640 pixels, therefore the width and height have the same ratio basis, and a coordinate component with a ratio of 0.5 corresponds to 320 pixels in the image.

[0086] It is understandable that size normalization is a necessary prerequisite for model inference. The improved RT-DETR object detection model is trained with a fixed input size. Scaling images of different sizes to be detected to a standard-sized input image ensures consistency in the model's input. The prediction head outputs normalized bounding box coordinates, facilitating model learning and prediction. When outputting the final object detection result, the normalized coordinates are multiplied by the image size to convert them back to absolute pixel coordinates, meeting the requirements for absolute measurement of target position and size in practical applications.

[0087] Referring to Figure 4, this is a chart comparing the number of channels in multi-scale feature maps of the RT-DETR object detection model, a professional analysis tool used in the feature extraction stage of deep learning models. The number of output channels of the lightweight backbone network increases as the feature map scale decreases (from 64 to 512), which conforms to the deep learning design logic of "small-scale feature maps corresponding to high-dimensional features." After cross-scale interaction, the number of channels in feature maps at all scales is unified to 256, achieving dimensional alignment of multi-scale features, which facilitates feature fusion in subsequent decoders. This chart intuitively demonstrates the functional effect of the "cross-scale interaction module." By unifying the number of channels, it not only preserves feature information at different scales but also reduces the complexity of subsequent calculations. This is a key design feature of the RT-DETR model in achieving efficient feature fusion in weakly supervised scenarios.

[0088] In one embodiment of the present invention, after obtaining the trained improved RT-DETR object detection model, the performance of the trained improved RT-DETR object detection model is evaluated using an additional validation dataset. The validation dataset is independent of the open-source dataset used for training; for example, the validation set portion of the PASCALVOC2012 dataset can be used, which contains 10,000 images and corresponding annotations. The evaluation process includes calculating the average precision metric of the trained improved RT-DETR object detection model on the validation dataset and comparing it with the average precision metric of the unimproved original RT-DETR model on the same validation dataset. When calculating the average precision metric, for each image in the validation dataset, forward inference is performed using the trained improved RT-DETR object detection model to obtain the object detection result, and the object detection result is compared with the ground truth annotations provided in the validation dataset. For each object category, the detection results are sorted according to the confidence score predicted by the model, the precision value at different recall rates is calculated, a precision-recall curve is plotted, and the area under the curve is calculated as the average precision for that category. The arithmetic mean of the average precision of all categories in the validation dataset is calculated to obtain the final average precision metric. Using the same validation dataset and evaluation code, inference and evaluation were performed on the unmodified original RT-DETR model, and the average accuracy metric of the unmodified original RT-DETR model on the validation dataset was calculated. The average accuracy metric calculated by the trained improved RT-DETR object detection model was then compared side-by-side with the average accuracy metric calculated by the unmodified original RT-DETR model, and the comparison results were presented in tabular or numerical list form.

[0089] In some embodiments, the evaluation process uses a standard Intersection over Union (IoU) threshold of 0.5 to calculate the average precision metric; that is, a prediction is considered correct when the IoU between the predicted bounding box and the ground truth bounding box is greater than or equal to 0.5. The validation dataset contains twenty object categories. The average precision for each category is calculated, and the mean of these twenty categories' average precision is then calculated as the model's final average precision metric. In some embodiments, in addition to using the IoU threshold of 0.5, the mean average precision is also calculated with IoU thresholds ranging from 0.5 to 0.95, with a step size of 0.05; this is a more comprehensive evaluation metric. The original RT-DETR model uses the same backbone network and number of training epochs as the improved RT-DETR object detection model, and is trained from scratch on the same training dataset to ensure fairness in the comparison. The evaluation process is conducted under the same hardware environment and software version to eliminate the impact of system differences on the evaluation results.

[0090] It's understandable that using an additional validation dataset for evaluation is to objectively measure the generalization performance of the improved RT-DETR object detection model after training. The validation dataset contains data not encountered during model training. Calculating mean precision (MP) is a common quantitative evaluation method in object detection, comprehensively considering both the model's prediction accuracy and recall. Comparing the MP with the unimproved original RT-DETR model on the same validation dataset aims to quantify the impact of the proposed improvements on the model's detection performance. Numerical comparisons directly reflect relative performance changes. The rigor of the evaluation process relies on maintaining consistency in the comparative experimental conditions, including using the same validation dataset, the same evaluation metric calculation method, the same training data foundation, and the same hyperparameter settings. Under such controlled conditions, observed differences in MP can be more reasonably attributed to improvements in model architecture and training strategies, rather than other confounding factors. Performance evaluation is a standard step in the model development process, used to verify the effectiveness of the method and provide direction for potential subsequent optimizations.

[0091] Referring to Figure 5, this is a comparison of the computational efficiency of different object detection models, focusing on the efficiency metrics of different models. This chart is used for efficiency evaluation in model selection. The improved RT-DETR achieves a balance between "high frame rate + low parameter count," making it more suitable for real-time detection needs in weakly supervised scenarios; while Faster R-CNN is more suitable for scenarios with low speed requirements and high accuracy requirements, providing a quantitative basis for model selection in different business scenarios. It intuitively demonstrates the optimization effect of the "improved RT-DETR" compared to the original RT-DETR. While ensuring controllable parameter count, it improves the frame rate and reduces computational load, proving the effectiveness of the pseudo-label improvement strategy in achieving a "performance-efficiency balance."

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0093] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A weakly supervised RT-DETR target detection method based on pseudo-labels, characterized in that, The method includes: acquiring an open-source dataset containing image data; performing weakly supervised annotation processing on the open-source dataset to generate initial pseudo-labels; and performing data augmentation operations on the image data and the initial pseudo-labels to obtain augmented training image data and corresponding augmented pseudo-labels; constructing an improved RT-DETR model architecture, the improved RT-DETR model architecture including a lightweight backbone network, a pseudo-label-guided cross-scale interaction module, and a decoder with a pseudo-label position constraint mechanism, including: constructing a lightweight backbone network as a feature extractor for multi-scale feature extraction of the input image and outputting a multi-scale feature map; connecting a pseudo-label-guided cross-scale interaction module after the multi-scale feature map output by the lightweight backbone network, the pseudo-label-guided cross-scale interaction module utilizing the input multi-scale feature map and the current training iteration. The process involves using enhanced pseudo-labels to calculate cross-scale feature fusion weights and generate a fused multi-scale interactive feature map. A decoder with a pseudo-label position constraint mechanism is constructed. This decoder receives the fused multi-scale interactive feature map and incorporates the spatial position information of the enhanced pseudo-labels during its attention calculation to generate decoded target query features. The enhanced training image data and corresponding enhanced pseudo-labels are input into the improved RT-DETR model architecture. An iterative pseudo-label update strategy is used to train the improved RT-DETR model architecture end-to-end, and the computational cost of forward and backward propagation is controlled in real-time during training to obtain a trained improved RT-DETR target detection model. The image to be detected is input into the trained improved RT-DETR target detection model to obtain the target detection result.

2. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 1, characterized in that, The open-source dataset is subjected to weakly supervised annotation to generate initial pseudo-labels. Data augmentation operations are then performed on the image data and the initial pseudo-labels to obtain augmented training image data and corresponding augmented pseudo-labels. This process includes: extracting image data and corresponding original annotation information from the open-source dataset, wherein the original annotation information includes the target category and bounding box coordinates; using a pre-trained general object detection model to infer from the image data, generating preliminary predicted boxes, and performing overlap-based matching and filtering on the preliminary predicted boxes and the original annotation information, retaining predicted boxes with confidence scores higher than a preset threshold as weakly supervised annotation results; converting the weakly supervised annotation results into annotation data consistent with the format of the original annotation information, as initial pseudo-labels; and simultaneously applying random horizontal flipping, random cropping, and color perturbation operations to the image data and the initial pseudo-labels to generate augmented training image data and corresponding augmented pseudo-labels.

3. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 2, characterized in that, The lightweight backbone network includes multiple convolutional stages, with the feature map output by each convolutional stage serving as a feature map at different scales, thus forming the multi-scale feature map. The pseudo-label-guided cross-scale interaction module utilizes the input multi-scale feature map and the enhanced pseudo-labels used in the current training iteration to calculate cross-scale feature fusion weights and generate a fused multi-scale interactive feature map. This includes: calculating the existence probability distribution of the target on different scale feature maps based on the size and position information of the target bounding box in the enhanced pseudo-labels; generating a guiding mask for each spatial location of each scale feature map based on the existence probability distribution; weighting the corresponding multi-scale feature map using the guiding mask; and generating the fused multi-scale interactive feature map through cross-scale feature addition and convolution operations.

4. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 3, characterized in that, The decoder with pseudo-label position constraint mechanism incorporates the spatial position information of the enhanced pseudo-labels during its attention calculation process to generate decoded target query features. This includes: calculating the cross-attention between the target query features and the fused multi-scale interactive feature map in each decoding layer of the decoder; generating a spatial position bias matrix based on the center point coordinates of all target instances in the enhanced pseudo-labels; adding the spatial position bias matrix element-wise to the cross-attention weight matrix to obtain a position-constrained attention weight matrix; using the position-constrained attention weight matrix to perform a weighted summation on the fused multi-scale interactive feature map to update the target query features; and outputting the final decoded target query features after iterative updates through multiple decoding layers.

5. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 4, characterized in that, An iterative pseudo-label update strategy is used to train the improved RT-DETR model architecture end-to-end, and the computational cost of forward and backward propagation is controlled in real time during training. This includes: dividing the enhanced training image data into multiple training batches; in the first training iteration, using the enhanced pseudo-labels as supervision signals to calculate the model prediction loss and perform backpropagation to update the model parameters; at the beginning of each subsequent training iteration, using the improved RT-DETR model architecture with the current parameter state to perform forward inference on the enhanced training image data to generate prediction results; performing non-maximum suppression processing on the prediction results, and selecting high-quality prediction boxes based on prediction confidence and overlap with the current enhanced pseudo-labels, using the high-quality prediction boxes to update the corresponding images' partial enhanced pseudo-labels, generating updated enhanced pseudo-labels; using the updated enhanced pseudo-labels as new supervision signals to calculate the model prediction loss and execute subsequent training iterations.

6. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 5, characterized in that, The computational cost of forward and backward propagation is controlled in real time during training, including: in the forward inference process of each training iteration, the intermediate feature maps in the improved RT-DETR model architecture are sampled with dynamic resolution, and the spatial resolution of the sampling is adjusted according to the preset computational cost budget; in the backward propagation process, only the gradients of some layers in the improved RT-DETR model architecture are calculated and backpropagated, and the contribution of the selected layers to the decrease of the loss function is dynamically selected according to their parameter updates.

7. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 6, characterized in that, The process of inputting the image to be detected into the trained improved RT-DETR object detection model to obtain object detection results includes: normalizing the size of the image to be detected to generate a standard-sized input image; inputting the standard-sized input image into the trained improved RT-DETR object detection model, extracting multi-scale feature maps through the lightweight backbone network, generating a fused multi-scale interactive feature map through the pseudo-label-guided cross-scale interaction module, and finally outputting the final target query feature by the decoder with a pseudo-label position constraint mechanism; inputting the final target query feature into a prediction head, which outputs the target class probability and bounding box coordinates as preliminary object detection results; and performing non-maximum suppression on the preliminary object detection results to eliminate redundant detection boxes and obtain the final object detection result.

8. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 7, characterized in that, The prediction head consists of two fully connected layer branches: one fully connected layer branch is used to predict the target class probability, and the other fully connected layer branch is used to predict the bounding box coordinates. The target class probability output by the prediction head is normalized by the softmax function, and the bounding box coordinates are normalized by the sigmoid function.

9. The weakly supervised RT-DETR target detection method based on pseudo-label improvement according to claim 8, characterized in that, The method further includes: after obtaining the trained improved RT-DETR object detection model, evaluating the performance of the trained improved RT-DETR object detection model using an additional validation dataset; the evaluation process includes calculating the average accuracy metric of the trained improved RT-DETR object detection model on the validation dataset, and comparing it with the average accuracy metric of the unimproved original RT-DETR model on the same validation dataset.

Images