A domain adaptive target detection method and system based on regression enhancement

Abstract

The present invention relates to a domain adaptive target detection method and system based on regression enhancement, the method comprising: obtaining a source domain data set; inputting the source domain data set into a student model, and performing strong and weak enhancement on the source domain data set to generate enhanced data; extracting features from the enhanced data to generate initial features; inputting the initial features into an RPN for classification and regression to generate a coarse positioning frame and a regression output value; dynamically assigning labels to the coarse positioning frame and screening coarse positioning frame positive samples; performing regression loss calculation on the coarse positioning frame positive samples to generate calculation results; performing target detection according to the calculation results to generate detection results. By replacing a fixed label allocation strategy with dynamic label allocation, the present application introduces a task alignment head in the RPN to encourage the model to make more consistent predictions, thereby enabling the model to retain high-quality detection frames through classification scores during prediction and reasoning.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a domain adaptive target detection method and system based on regression enhancement. Background Art

[0002] Domain adaptive object detection aims to train a detector with labeled source domain data and generalize it to the unlabeled target domain. However, due to the shift between domains, the performance of the detector degrades significantly. The inherent challenges of this task can be summarized into three aspects. First, the challenge of marginal distribution difference stems from the differences in angles, lighting conditions, weather, etc. between the target domain and source domain data, which greatly affects the robustness of the model. Second, the data challenge stems from the lack of labeled data in the target domain, which makes it impossible for the model to obtain accurate labels for effective target domain training. Finally, the challenge of task difference stems from the coexistence of two tasks, classification and regression, in object detection. How to bridge the gap between these two tasks in the presence of domain difference remains an unsolved problem.

[0003] Nowadays, many methods have been proposed to alleviate the problem of detector performance degradation caused by domain shift. In response to the challenge of distribution difference, adversarial learning methods are introduced in the prior art to align the feature-level distribution of images, while some other methods propose image transformation methods to align the hierarchical distribution of images. When dealing with the challenge of missing labeled data, some researchers use self-training methods to provide pseudo labels for the target domain. In order to address the differences between the classification and regression tasks of single-stage detectors, task alignment heads are also proposed to make the predictions between the two tasks more consistent. Other methods introduce optimal transfer methods to assign positive and negative samples for fully supervised object detection tasks. However, it should be noted that these methods are mainly designed for traditional object detection tasks and single-stage object detection frameworks. Few methods consider the misalignment problem between classification and localization in domain adaptive object detection and two-stage detection frameworks.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology does not consider the localization quality in the task of domain adaptive object detection. However, due to the difference in domains, the localization task in domain adaptive object detection is more challenging. In the field of domain adaptive object detection, there is still a research gap in the method of solving the inconsistency of subtasks (or).

[0006] In order to solve the above technical problems, a first aspect of the present invention provides a domain adaptive target detection method based on regression enhancement, the method comprising:

[0007] Acquire a source domain data set; the source domain data is image data;

[0008] Inputting the source domain data set into the student model, and performing strong and weak enhancement on the source domain data set to generate enhanced data;

[0009] Performing feature extraction on the enhanced data to generate initial features;

[0010] Input the initial features into RPN for classification and regression to generate a rough positioning frame and regression output value;

[0011] Dynamically assigning labels to the coarse positioning frame and screening positive samples of the coarse positioning frame;

[0012] Performing regression loss calculation on the positive samples of the coarse positioning frame to generate a calculation result;

[0013] Target detection is performed according to the calculation result to generate a detection result.

[0014] In one embodiment of the present invention, the steps of inputting the initial features into the RPN for classification and regression, and generating a rough positioning frame and a regression output value include:

[0015] Classifying the initial features according to the task alignment head to generate classification features;

[0016] Regressing the initial features according to the task alignment head to generate regression features;

[0017] Combining the classification feature with the regression feature to generate a combined feature;

[0018] A regression output value is generated according to the combined features.

[0019] In one embodiment of the present invention, the calculation formula of the regression output value is as follows:

[0020]

[0021] in, Table feature graph The output result after the i-th classification convolution layer is: Table feature graph The output result after the i-th regression convolution layer is: express and splicing features.

[0022] In one embodiment of the present invention, the step of dynamically assigning labels to the coarse positioning frame and generating a positive sample of the coarse positioning frame includes:

[0023] Dynamically assigning labels to the coarse positioning frame to generate positive samples of initial coarse positioning frame;

[0024] The initial coarse positioning frame positive sample is dynamically IOU-screened to generate a coarse positioning frame positive sample.

[0025] In one embodiment of the present invention, the IOU threshold calculation formula for the dynamic IOU screening is as follows:

[0026]

[0027] in, is assigned to the ground-truth bounding box The candidate box, is a hyperparameter that controls the threshold update speed; is the IOU threshold.

[0028] In one embodiment of the present invention, the step of dynamically assigning labels to the coarse positioning frame to generate positive samples of the initial coarse positioning frame includes:

[0029] Generate the real bounding box and dynamic candidate box based on the rough positioning box;

[0030] Calculating a matching cost between the real bounding box and the dynamic candidate box;

[0031] An initial rough positioning box positive sample is generated according to the matching cost.

[0032] In one embodiment of the present invention, the calculation formula of the matching cost is as follows:

[0033]

[0034] in, Item represents a candidate box With the ground-truth bounding box The IOU matching cost between them; Represents the difference matching cost between the candidate box and the true bounding box shape; Represents the distance matching cost between the candidate box and the true bounding box; represents the classification matching cost.

[0035] A second aspect of the present invention provides a domain adaptive target detection system based on regression enhancement, which is applied to a method proposed in any one of the first aspects above, wherein the system comprises: a data acquisition module, a feature extraction module, a first calculation module, and a second calculation module;

[0036] The data acquisition module is configured to: acquire a source domain dataset;

[0037] The feature extraction module is configured to: input the source domain data set into the student model, and perform strong and weak enhancement on the source domain data set to generate enhanced data; perform feature extraction on the enhanced data to generate initial features;

[0038] The first calculation module is configured to: input the initial features into the RPN for classification and regression, generate a rough positioning frame and a regression output value; dynamically assign labels to the rough positioning frame, and screen positive samples of the rough positioning frame;

[0039] The second calculation module is configured to: perform regression loss calculation on the positive sample of the coarse positioning frame to generate a calculation result; perform target detection according to the calculation result to generate a detection result.

[0040] The third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect or any possible implementation of the first aspect is implemented.

[0041] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or any possible implementation of the first aspect.

[0042] The above technical solution of the present invention has the following advantages compared with the prior art:

[0043] The domain adaptive object detection method and system based on regression enhancement described in the present invention replaces the fixed label allocation strategy with dynamic label allocation. Taking into account the ambiguity of the relationship between classification and regression prediction in the Region Proposal Network (RPN), we introduce a task alignment head and fuzzy penalty loss in the RPN to encourage the model to make more consistent predictions, thereby enabling the model to retain high-quality detection boxes through classification scores during prediction and reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0045] Figure 1 It is a flowchart of a domain adaptive target detection method and system based on regression enhancement provided by the present invention;

[0046] Figure 2It is a scatter plot of a baseline model in a domain adaptive target detection method based on regression enhancement and a system provided by the present invention;

[0047] Figure 3 It is a scatter plot of a domain adaptive target detection method based on regression enhancement and a regression enhancement model in the system provided by the present invention;

[0048] Figure 4 A system architecture diagram of a domain adaptive target detection method and system based on regression enhancement provided by the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0050] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0051] Reference Figure 1 As shown, the present invention provides a domain adaptive target detection method based on regression enhancement, the method comprising:

[0052] S100, obtaining a source domain dataset;

[0053] In step S100, the source domain dataset obtained is ;in, represents the source domain image, represents the source domain category, Represents the source domain coordinate information.

[0054] S200, inputting the source domain data set into the student model, and performing strong and weak enhancement on the source domain data set to generate enhanced data;

[0055] In step S200, the source domain dataset is obtained Input to the student model for supervised training and use the source domain dataset Strong and weak enhancements are performed, and the results after strong and weak enhancements are enhanced data.

[0056] S300, extracting features from the enhanced data to generate initial features;

[0057] In step S300, the enhanced data is subjected to feature extraction through Backbone. Specifically, the underlying backbone network is used to extract features from the image, and we obtain a feature map Among them, feature extraction is the basis of computer vision tasks, and a good feature extraction network can significantly improve the performance of the algorithm. In computer vision tasks, the network that extracts features from images is called the backbone network.

[0058] S400, inputting the initial features into RPN for classification and regression, generating a rough positioning frame and regression output value;

[0059] In step S400, the initial features are input into the RPN for classification and regression, and the steps of generating a rough positioning frame and a regression output value include: classifying the initial features according to the task alignment head to generate classification features; regressing the initial features according to the task alignment head to generate regression features; combining the classification features and the regression features to generate combined features; and generating a regression output value according to the combined features. The calculation formula of the regression output value is as follows:

[0060]

[0061] in, Table feature graph The output result after the i-th classification convolution layer is: Table feature graph The output result after the i-th regression convolution layer is: express and splicing features.

[0062] In practical application scenarios, considering the inherent ambiguity of the association between RPN classification and regression predictions, we introduce a task alignment head and ambiguity penalty loss to encourage the model to make more consistent predictions. We insert convolutional layers to obtain feature maps specific to classification and regression tasks respectively. and ; In order to strengthen the interdependence between subtasks, the convolutional layer Join features Finally, through the deformable convolution layer The fused features are transferred to generate regression predictions (including the center point of the candidate box and the width and height offsets) to obtain the regression output value.

[0063] In addition, we observe that there are a large number of candidate boxes in RPN with objectness scores close to 0.5, which indicates that the model has difficulty in determining whether these candidate boxes are foreground or background. Therefore, we propose the following loss function to encourage the model to generate more confident predictions:

[0064]

[0065] in, express The class probability of the output after the classification head and sigmoid activation. When the predicted probability is close to 0, it means that the model believes that the area is likely to contain no object, and the loss function decreases. Similarly, when the predicted probability is close to 1, it means that the model believes that the area is likely to contain an object, and the loss function also decreases. Through formula (2), the model is forced to reduce the uncertainty of the prediction.

[0066] S500, dynamically assigning labels to the coarse positioning frame, and screening positive samples of the coarse positioning frame;

[0067] In step S500, the steps of dynamically assigning labels to the coarse positioning frame and generating a coarse positioning frame positive sample include: dynamically assigning labels to the coarse positioning frame to generate an initial coarse positioning frame positive sample; and dynamically screening the initial coarse positioning frame positive sample to generate a coarse positioning frame positive sample. The calculation formula for the dynamic IOU screening is as follows:

[0068] (3)

[0069] in, is assigned to the ground-truth bounding box The candidate box, is a hyperparameter that controls the threshold update speed. The steps of dynamically assigning labels to the coarse positioning frame and generating initial coarse positioning frame positive samples include: generating a true bounding box and a dynamic candidate frame according to the coarse positioning frame; calculating the matching cost between the true bounding box and the dynamic candidate frame; and generating an initial coarse positioning frame positive sample according to the matching cost. The matching cost calculation formula is as follows:

[0070]

[0071] in, Item represents a candidate box With the ground-truth bounding box The IOU matching cost between them.

[0072] In actual application scenarios, the fixed label allocation strategy is replaced by the dynamic label allocation of the optimal transmission, and the dynamic label allocation strategy helps the network find the optimal positive and negative samples. Specifically, first use formula (5) to determine each true bounding box The number of dynamic candidate boxes :

[0073]

[0074] in, is a hyperparameter, indicating the number of candidate boxes used to calculate the number of dynamic candidate boxes. Indicates allocation to The i-th candidate box, The IOU between them.

[0075] Then, each candidate box is calculated and the ground-truth bounding box The matching cost between them is shown in formula (6):

[0076]

[0077] in, Item represents a candidate box With the ground-truth bounding box The IOU matching cost between is specifically defined as:

[0078]

[0079] In addition, we use the model's confidence in the bounding box classification to represent the classification matching cost :

[0080]

[0081] here, Cross entropy, is the class probability of the feature output after the classification head and sigmoid activation, The category of the true bounding box or pseudo-labeled box.

[0082] Secondly, unlike OTA, this application is an anchor-based object detection framework, and the shapes of the candidate boxes and the true bounding boxes are also different. Therefore, this application additionally introduces the difference matching cost between the candidate boxes and the true bounding boxes. , specifically defined as:

[0083]

[0084] in, represents the inverse tangent function, and Represent the true bounding box The width and height of and Represents a candidate box The width and height of the .

[0085] Finally, this application considers the distance matching cost between the candidate box and the true bounding box Unlike OTA which considers the Euclidean distance between feature points, we consider the distance between the center point of the candidate box and the bounding box as the matching cost, which is defined as follows:

[0086]

[0087] in, and Represent the center coordinates of the candidate box and the true bounding box respectively.

[0088] The matching cost is calculated based on each true bounding box and all candidate boxes , and the matching cost is the smallest The coarse positioning box will be used as a coarse positioning box positive sample to participate in the regression loss calculation, and the remaining samples will be used as negative samples to participate only in the classification loss calculation.

[0089] This application proposes a dynamic IOU threshold mechanism for the ROI branch, which calculates the IOU between the candidate box and the real bounding box in real time, adaptively adjusts the IOU threshold, better balances the selection of positive and negative samples, and further improves the performance of the model on regression tasks. As shown in formula (3), initially, is set to 0.4. At the beginning of training, a relatively low threshold may select more candidate boxes as positive samples, allowing the model to learn more sample features. As training progresses, It gradually increases, indicating that the screening criteria for candidate boxes assigned to the true bounding box are becoming more and more stringent.

[0090] S600, performing regression loss calculation on the positive sample of the rough positioning frame to generate a calculation result;

[0091] In step S600, the matching cost between the real bounding box and the dynamic candidate box is calculated , and the matching cost The smallest front The dynamic candidate boxes are taken as the positive samples of the coarse positioning boxes, and the regression loss is calculated for them to generate the corresponding calculation results.

[0092] S700: Perform target detection according to the calculation result to generate a detection result.

[0093] In step S700, target detection is performed on the calculation results generated above to determine the specific type of the current target, such as whether the current target is a car or a person, and then generate the corresponding detection result. At this time, the student model converges, and the data of the student model is copied to the teacher model to obtain the target data set. The target data set is , represents the target domain image. The target domain has no labeled information and therefore does not contain category and coordinate information. The target dataset is input into the teacher model. During the training process, the detection box generated by the teacher model is recorded as a pseudo label. , Represents classification and coordinate information respectively.

[0094] The pseudo labels are fed into the student model, which is trained using these pseudo labels while updating the weights of the teacher model using the exponential moving average algorithm (EMA):

[0095]

[0096] in, represents the teacher model weight parameter, represents the student model weight parameter, It is a hyperparameter that controls the update speed of the teacher model weights.

[0097] The overall loss of the teacher model can be expressed as:

[0098]

[0099] Reference Figure 2 and Figure 3 As shown in the figure, there is no obvious correlation between the classification quality (classification score) and regression quality (IoU) of the baseline model, while there is a correlation in the regression enhancement model of this application, which is beneficial for the model to retain high-quality detection boxes according to the classification scores during reasoning, non-maximum suppression, and pseudo-label screening, thereby improving model performance.

[0100] in, and They are the losses of RPN and ROI branches in Faster RCNN respectively. is a hyperparameter that balances the total loss.

[0101] Second, refer to Figure 4 As shown, the present application provides a domain adaptive target detection system based on regression enhancement, which is applied to a method proposed in any one of the first aspects above, and the system includes: a data acquisition module 100, a feature extraction module 200, a first calculation module 300, and a second calculation module 400;

[0102] The data acquisition module 100 is configured to: acquire a source domain dataset;

[0103] The feature extraction module 200 is configured to: input the source domain data set into the student model, and perform strong and weak enhancement on the source domain data set to generate enhanced data; perform feature extraction on the enhanced data to generate initial features;

[0104] The first calculation module 300 is configured to: input the initial features into the RPN for classification and regression, generate a rough positioning frame and a regression output value; dynamically assign labels to the rough positioning frame, and screen positive samples of the rough positioning frame;

[0105] The second calculation module 400 is configured to: perform regression loss calculation on the positive sample of the coarse positioning frame to generate a calculation result; perform target detection according to the calculation result to generate a detection result.

[0106] The effects of the above-mentioned system when applying the above-mentioned method can be found in the description of the above-mentioned method embodiment, which will not be repeated here.

[0107] The third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect or any possible implementation of the first aspect is implemented.

[0108] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or any possible implementation of the first aspect.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A domain adaptive object detection method based on regression enhancement, characterized in that: The method comprises: Acquire a source domain data set; the source domain data is image data; Inputting the source domain data set into the student model, and performing strong and weak enhancement on the source domain data set to generate enhanced data; Performing feature extraction on the enhanced data to generate initial features; Input the initial features into RPN for classification and regression to generate a rough positioning frame and regression output value; Dynamically assigning labels to the coarse positioning frame and screening positive samples of the coarse positioning frame; Performing regression loss calculation on the positive samples of the coarse positioning frame to generate a calculation result; Perform target detection according to the calculation result to generate a detection result; The steps of dynamically assigning labels to the rough positioning frame and generating a positive sample of the rough positioning frame include: Dynamically assigning labels to the coarse positioning frame to generate positive samples of initial coarse positioning frame; Performing dynamic IOU screening on the initial coarse positioning frame positive samples to generate coarse positioning frame positive samples; The IOU threshold calculation formula for the dynamic IOU screening is as follows: in, is assigned to the ground-truth bounding box The candidate box, is a hyperparameter that controls the threshold update speed; is the IOU threshold; The steps of dynamically assigning labels to the coarse positioning frame to generate positive samples of the initial coarse positioning frame include: Generate the real bounding box and dynamic candidate box based on the rough positioning box; Calculating a matching cost between the real bounding box and the dynamic candidate box; Generate an initial rough positioning frame positive sample according to the matching cost; The calculation formula of the matching cost is as follows: in, Item represents a candidate box With the ground-truth bounding box The IOU matching cost between them; Represents the difference matching cost between the candidate box and the true bounding box shape; Represents the distance matching cost between the candidate box and the true bounding box; represents the classification matching cost.

2. The domain adaptive target detection method based on regression enhancement according to claim 1, characterized in that: The steps of inputting the initial features into the RPN for classification and regression, and generating a rough positioning frame and regression output value include: Classifying the initial features according to the task alignment head to generate classification features; Regressing the initial features according to the task alignment head to generate regression features; Combining the classification feature with the regression feature to generate a combined feature; A regression output value is generated according to the combined features.

3. The domain adaptive target detection method based on regression enhancement according to claim 2, characterized in that: The calculation formula of the regression output value is as follows: in, Table feature graph The output result after the i-th classification convolution layer is: Table feature graph The output result after the i-th regression convolution layer is: express and splicing features.

4. A domain adaptive object detection system based on regression enhancement, characterized in that: A domain adaptive target detection method based on regression enhancement applied to any one of claims 1 to 3 above, the system comprising: a data acquisition module, a feature extraction module, a first calculation module, and a second calculation module; The data acquisition module is configured to: acquire a source domain dataset; The feature extraction module is configured to: input the source domain data set into the student model, and perform strong and weak enhancement on the source domain data set to generate enhanced data; perform feature extraction on the enhanced data to generate initial features; The first calculation module is configured to: input the initial features into the RPN for classification and regression, generate a rough positioning frame and a regression output value; dynamically assign labels to the rough positioning frame, and screen positive samples of the rough positioning frame; The second calculation module is configured to: perform regression loss calculation on the positive sample of the coarse positioning frame to generate a calculation result; perform target detection according to the calculation result to generate a detection result.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the domain adaptive target detection method based on regression enhancement as described in any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a domain adaptive target detection method based on regression enhancement as claimed in any one of claims 1 to 3 are implemented.

Images