Domain adaptive target detection method and device in extreme scenarios

Abstract

This application discloses a domain-adaptive target detection method and device for extreme scenarios. The method includes: obtaining multiple source domain images with artificial labels, where the source domain images are non-extreme scene images; obtaining multiple target domain images based on the multiple source domain images, where the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images are unlabeled; performing model training and label generation processing based on the multiple source domain images and the multiple target domain images to obtain a first detection model and a target domain image set, where the target domain image set includes multiple target domain images with pseudo labels; training the first detection model based on the target domain image set to obtain a second detection model; and performing target detection on the image to be tested based on the second detection model to obtain a target detection result for the image to be tested. This application can effectively improve the accuracy of target detection for extreme scenarios and can be widely used in the field of computer vision technology.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a domain-adaptive target detection method and device in extreme scenarios. Background Art

[0002] The task of object detection aims to automatically identify and locate specific types of objects within a given image or video frame. Currently, object detection models are typically trained using manually labeled extreme scene images, and the trained models are then used to perform extreme scene object detection tasks. However, because extreme scene images are often difficult to obtain, the limited number of extreme scene image samples leads to insufficient training of object detection models, which in turn affects their performance in extreme scene object detection tasks. Consequently, the accuracy of object detection for extreme scenes needs to be improved. Summary of the Invention

[0003] The embodiments of the present application provide a domain-adaptive target detection method and device for extreme scenarios, which are used to improve the target detection accuracy for extreme scenarios.

[0004] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0005] On the one hand, embodiments of the present application provide a domain-adaptive object detection method for extreme scenarios, comprising the following steps:

[0006] Acquire a plurality of source domain images with manual labels; wherein the source domain images are non-extreme scene images;

[0007] A plurality of target domain images are obtained based on the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels;

[0008] Performing model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes the plurality of target domain images with pseudo labels;

[0009] Training the first detection model according to the target domain image set to obtain a second detection model;

[0010] Object detection is performed on the image to be tested according to the second detection model to obtain an object detection result of the image to be tested.

[0011] Furthermore, in one embodiment, obtaining a plurality of target domain images based on the plurality of source domain images includes:

[0012] The image generation processing of the extreme scene is performed based on each of the source domain images to obtain the target domain image corresponding to each of the source domain images, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining a plurality of target domain images.

[0013] Furthermore, in one embodiment, performing model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set includes:

[0014] Performing inference processing on the plurality of target domain images based on a preset teacher network to obtain a plurality of initial candidate frames for each of the target domain images;

[0015] Obtaining the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image;

[0016] Training a preset student network according to the i-th batch of training sets; wherein the i-th batch of training sets includes a plurality of the target domain images with the pseudo labels and the source domain images belonging to the i-th batch and with the artificial labels;

[0017] Updating the model parameters of the student network using the supervised loss and unsupervised loss of the student network;

[0018] When i<N, N is a first preset threshold. If the i-th batch does not meet the preset update condition, then set i=i+1, and return to the step of training the preset student network according to the training set of the i-th batch; otherwise, update the model parameters of the teacher network, and return to the step of performing inference processing on the plurality of target domain images based on the preset teacher network until i≥N, obtain the target domain image set according to each target domain image and its pseudo label, and determine the student network after the model parameters are updated as the first detection model;

[0019] The above steps start from i=1.

[0020] Furthermore, in one embodiment, obtaining the pseudo label of each target domain image according to the plurality of initial candidate boxes of each target domain image includes:

[0021] Performing screening processing on the plurality of initial candidate frames of the target domain image to obtain a plurality of target candidate frames of the target domain image;

[0022] Performing voting on each of the target candidate boxes in the target domain image to obtain a voting weight for each of the target candidate boxes in the target domain image;

[0023] The pseudo label of the target domain image is obtained according to the coordinate position and voting weight of each target candidate box of the target domain image.

[0024] Furthermore, in one embodiment, updating the model parameters of the student network using the supervised loss and the unsupervised loss of the student network includes:

[0025] Obtaining a supervision loss of the student network based on the predicted labels of the source domain images belonging to the i-th batch and the manual labels; wherein the predicted labels of the source domain images are label information obtained by predicting the source domain images by the student network;

[0026] Obtaining an unsupervised loss of the student network based on the predicted label of each target domain image and the pseudo label; wherein the predicted label of the target domain image is label information obtained by predicting the target domain image by the student network;

[0027] Obtaining a target loss of the student network according to the supervised loss and the unsupervised loss of the student network;

[0028] Backpropagation is performed using the target loss of the student network to update the model parameters of the student network.

[0029] Furthermore, in one embodiment, updating the model parameters of the teacher network includes:

[0030] According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

[0031] Furthermore, in one embodiment, the training of the first detection model based on the target domain image set to obtain the second detection model includes:

[0032] Obtaining a third detection model according to the first detection model;

[0033] Training the third detection model according to the j-th batch of training sets; wherein the j-th batch of training sets includes the target domain images belonging to the j-th batch and having the pseudo labels;

[0034] Updating model parameters of the third detection model using the unsupervised loss of the third detection model;

[0035] When j<N', N' is a second preset threshold, and j=j+1 is set, and the process returns to the step of training the third detection model according to the j-th batch of training sets until j≥N', and the third detection model after the model parameters are updated is determined as the second detection model;

[0036] The above steps start from j=1.

[0037] Furthermore, in one embodiment, the second detection model includes an input layer, a feature extraction structure, and a prediction head, wherein:

[0038] The input layer is used to perform block processing on the image to be tested to obtain an image block sequence;

[0039] The feature extraction structure includes a plurality of sequentially connected feature extraction modules, the input features of the first feature extraction module are the image block sequence, and the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to perform feature extraction on the input features of the feature extraction module to obtain the search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain the output features of the feature extraction module based on the search features and the input features of the feature extraction module;

[0040] The prediction head is used to perform prediction based on the output features of the last feature extraction module to obtain the target detection result of the image to be tested.

[0041] Furthermore, in one embodiment, obtaining the output feature of the feature extraction module based on the search feature and the input feature of the feature extraction module includes:

[0042] Obtaining a fusion feature of the feature extraction module according to the search feature and the input feature of the feature extraction module;

[0043] Obtaining a dynamic prompt of the feature extraction module according to the search feature and the input feature of the feature extraction module;

[0044] The output feature of the feature extraction module is obtained according to the dynamic prompt and the search feature of the feature extraction module.

[0045] On the other hand, an embodiment of the present application provides a domain-adaptive object detection device for extreme scenarios, including:

[0046] An acquisition module, configured to acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images;

[0047] A first processing module is configured to obtain a plurality of target domain images based on the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels;

[0048] a second processing module, configured to perform model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels;

[0049] a third processing module, configured to train the first detection model according to the target domain image set to obtain a second detection model;

[0050] The fourth processing module is used to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0051] According to the domain adaptive target detection method and device in extreme scenarios of the embodiment of the present application, first, multiple target domain images are generated by using multiple source domain images with artificial labels, the source domain images are non-extreme scene images, and the target domain images are extreme scene images, and only the first target domain image has an artificial label, and the remaining target domain images are unlabeled, which can effectively increase the number of target domain image samples and improve the diversity of target domain images; then, model training and label generation processing are performed based on the multiple source domain images and the multiple target domain images, aiming to generate corresponding pseudo labels for each target domain image, and supervised learning and unsupervised learning are performed on the detection model used to perform target detection tasks in extreme scenarios, thereby obtaining a first detection model and A target domain image set, wherein the target domain image set includes multiple target domain images with pseudo labels, which not only improves the integrity of the target domain images and reduces the label annotation cost, but also ensures that the detection model is adequately trained, reduces the possibility of problems such as overfitting, and improves the robustness of the detection model; then, the target domain image set is used to train the first detection model, so that the detection model further learns the relationship between the target domain image and its pseudo label, and then obtains the second detection model; finally, target detection is performed on the image to be tested according to the second detection model to obtain the target detection result of the image to be tested, thereby realizing the target detection task of extreme scenes. In this way, the embodiment of the present application can effectively improve the target detection accuracy for extreme scenes.

[0052] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of the domain adaptive object detection method for extreme scenarios provided by this application;

[0054] Figure 2 This is a schematic diagram of the domain adaptive target detection method for extreme scenarios provided by this application;

[0055] Figure 3 This is a structural diagram of the domain adaptive target detection device for extreme scenarios provided by this application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0060] The task of object detection aims to automatically identify and locate specific types of objects within a given image or video frame. Currently, object detection models are typically trained using manually labeled extreme scene images, and the trained models are then used to perform extreme scene object detection tasks. However, because extreme scene images are often difficult to obtain, the limited number of extreme scene image samples leads to insufficient training of object detection models, which in turn affects their performance in extreme scene object detection tasks. Consequently, the accuracy of object detection for extreme scenes needs to be improved.

[0061] In view of this, embodiments of the present application provide a domain-adaptive target detection method and device for extreme scenarios, aiming to improve the target detection accuracy for extreme scenarios.

[0062] First, the implementation steps of the domain adaptive object detection method for extreme scenarios provided by this application will be described in detail below with reference to the accompanying drawings.

[0063] The domain-adaptive target detection method for extreme scenarios provided in the embodiments of the present application can be applied to a terminal or a server, or can be software running on a terminal or server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to such. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to such. Blockchain is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0064] Reference Figure 1 , Figure 1 4 is a flowchart of a domain adaptive target detection method for extreme scenarios provided by the present application. The domain adaptive target detection method may include the following steps S100-S500.

[0065] S100, obtaining multiple source domain images with manual labels.

[0066] It should be noted that manual labels refer to label information obtained by manually annotating images. The label information can be flexibly set according to actual conditions. For example, the label information may include at least one of the location or type of the area of ​​interest in the image, but is not limited to this.

[0067] It is understood that the source domain refers to non-extreme scenes, and further, the source domain images refer to non-extreme scene images, that is, images associated with non-extreme scenes. It is worth noting that the non-extreme scenes in the embodiments of the present application are daytime scenes, that is, the source domain images refer to daytime scene images.

[0068] In this step, first, multiple initial source domain images are obtained; then, corresponding label information is annotated for each source domain image through manual labeling, so that each source domain image has a corresponding manual label, thereby obtaining multiple source domain images with manual labels.

[0069] S200 , obtaining multiple target domain images based on multiple source domain images.

[0070] It should be noted that only the first target domain image has a manual label, and the remaining target domain images have no labels.

[0071] It is understood that the target domain refers to extreme scenes, and further, the target domain images refer to extreme scene images, that is, images associated with extreme scenes. It is worth noting that the extreme scenes in the embodiments of the present application are foggy scenes and nighttime scenes, and further, the target domain images include nighttime scene images and foggy scene images.

[0072] In this step, after obtaining multiple source domain images, image generation processing is performed on the multiple source domain images so that each source domain image is converted into a corresponding target domain image, thereby obtaining multiple target domain images. This effectively increases the number of target domain image samples and improves the diversity of target domain images. Only the first target domain image has a manual label, and the remaining target domain images are unlabeled.

[0073] S300 , performing model training and label generation processing based on multiple source domain images and multiple target domain images to obtain a first detection model and a target domain image set.

[0074] It should be noted that the target domain image set may include multiple target domain images with pseudo labels, where the pseudo label refers to label information obtained by predicting the target domain image through a certain network.

[0075] In this step, cross-domain model training and label generation processing are performed based on multiple source domain images and multiple target domain images. The implementation of this processing mainly relies on a preset teacher network and a preset student network, wherein the teacher network is used to generate corresponding pseudo labels for each target domain image and regularly update the generated pseudo labels, while the student network is used to iteratively train based on source domain images with artificial labels and target domain images with pseudo labels and regularly update its own model parameters and the model parameters of the teacher network. Through the synergy of the teacher network and the student network, cross-domain knowledge transfer and pseudo label generation of target domain images can be achieved. The first detection model and the target domain image set are obtained through the above processing, wherein the first detection model is the iteratively trained student network, and the target domain image set includes multiple target domain images with pseudo labels generated by the teacher network.

[0076] S400: Training the first detection model according to the target domain image set to obtain a second detection model.

[0077] It should be noted that the second detection model refers to a model suitable for target detection tasks in extreme scenarios.

[0078] In this step, after obtaining multiple target domain images with pseudo labels, the first detection model is trained using these target domain images with pseudo labels so that the detection model has the ability to perform target detection tasks in extreme scenarios, thereby obtaining the second detection model.

[0079] S500 , performing target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0080] It should be noted that the image to be tested refers to an image to be tested and associated with an extreme scene, such as an image to be tested of a night scene or an image to be tested of a foggy scene.

[0081] It can be understood that the target detection result refers to the result obtained by performing target detection on the image to be tested, and its type can be flexibly set according to actual conditions. For example, the target detection result may include at least one of the location or type of the area of ​​interest of the image to be tested, but is not limited to this.

[0082] In this step, the image to be tested is input into the second detection model, and prediction processing is performed by the second detection model to obtain the target detection result of the image to be tested, thereby achieving the target detection task in extreme scenes.

[0083] It can be seen that in the embodiment of the present application, first, multiple target domain images are generated by using multiple source domain images with artificial labels, the source domain images are non-extreme scene images, and the target domain images are extreme scene images, and only the first target domain image has an artificial label, and the remaining target domain images are unlabeled, which can effectively increase the number of target domain image samples and improve the diversity of target domain images; then, model training and label generation processing are performed based on the multiple source domain images and the multiple target domain images, aiming to generate corresponding pseudo labels for each target domain image, and supervised learning and unsupervised learning are performed on the detection model used to perform target detection tasks in extreme scenes, thereby obtaining A first detection model and a target domain image set, wherein the target domain image set includes multiple target domain images with pseudo labels, which not only improves the integrity of the target domain images and reduces the label annotation cost, but also ensures that the detection model is adequately trained, reduces the possibility of problems such as overfitting, and improves the robustness of the detection model; then, the first detection model is trained using the target domain image set so that the detection model further learns the relationship between the target domain images and their pseudo labels, thereby obtaining a second detection model; finally, target detection is performed on the image to be tested according to the second detection model to obtain the target detection result of the image to be tested, thereby achieving the target detection task for extreme scenes. In this way, the embodiment of the present application can effectively improve the target detection accuracy for extreme scenes.

[0084] The above steps will be further described below.

[0085] In some embodiments, reference Figure 2 In the above step S200, the process of obtaining multiple target domain images based on multiple source domain images may include:

[0086] Based on each source domain image, image generation processing of extreme scenes is performed to obtain the target domain image corresponding to each source domain image, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining multiple target domain images.

[0087] In this embodiment, each source domain image is first processed for extreme scene image generation based on that source domain image to obtain a target domain image corresponding to that source domain image. By traversing all source domain images, target domain images corresponding to all source domain images can be obtained. It is understood that target domain images refer to extreme scene images, which can include nighttime and foggy scene images. Then, only the manual label of the first source domain image is assigned to the target domain image corresponding to that first source domain image, while the target domain images corresponding to the remaining source domain images are not labeled. In this way, multiple target domain images can be obtained.

[0088] Specifically, the above image generation process is divided into two parts: foggy scene image generation and nighttime scene image generation.

[0089] In terms of foggy scene image generation, a depth estimation model is used to obtain the depth information of the source domain image. Specifically, the source domain image is input into the depth estimation model, and the high-dimensional features of the source domain image are extracted through the encoder of the depth estimation model. The high-dimensional features are mapped into a depth prediction map as depth information through the decoder of the depth estimation model. After obtaining the depth information of the source domain image, the source domain image and its depth information are synthesized into a foggy image based on a preset fog scattering model to obtain the foggy scene image corresponding to the source domain image. This can improve the generation quality of foggy scene images.

[0090] Optionally, the depth estimation model can be configured according to actual conditions, and this embodiment does not impose any specific limitation thereto.

[0091] For example, the above-mentioned depth estimation model can be depthanything, that is, the source domain image is input into the ViT-based DINOv2 encoder of depthanything to extract high-dimensional features, and the high-dimensional features are passed to the DPT-based decoder of depthanything, which maps the features into a depth prediction map, that is, depth information.

[0092] Optionally, the fog scattering model follows the following formula (1):

[0093] I(x)=J(x)t(x)+A(1-t(x)) (1);

[0094] In formula (1), I(x) represents the pixel value of the foggy scene image corresponding to the source domain image; J(x) represents the pixel value of the source domain image; A represents the atmospheric light value; t(x) represents the transmittance of the source domain image, which is defined as the light transmission ratio from the scene point to the camera. The relationship between the transmittance and the depth information of the source domain image follows the following formula (2):

[0095] t(x)=e -βd (x) (2);

[0096] In formula (2), d(x) represents the depth information of the source image; β represents the preset medium scattering coefficient. The larger the value of the medium scattering coefficient, the higher the fog density of the foggy scene image corresponding to the source image. It is understandable that, depending on actual needs, the medium scattering coefficients corresponding to different source images can be the same or different, and can be set according to actual conditions.

[0097] In terms of night scene image generation, a generative adversarial network is used to convert the source domain image into the night scene image corresponding to the source domain image, which can improve the generation quality of night scene images.

[0098] Optionally, the above-mentioned generative adversarial network can be set according to actual conditions, and this embodiment does not specifically limit this.

[0099] For example, the above-mentioned generative adversarial network can be CycleGAN, which uses unsupervised learning and combines pairwise adversarial loss and cycle consistency loss to achieve high-quality image translation between different domains. Specifically, CycleGAN can include a generator G, a generator F, a discriminator DX, and a discriminator DY. Generator G is used to convert source domain images into target domain images, generator F is used to convert target domain images into source domain images, and discriminator DX and discriminator DY are both used to determine whether the input image is real. Discriminator DX is targeted at the source domain, and discriminator DY is targeted at the target domain. The goal of CycleGAN is to achieve training through adversarial loss and cycle consistency loss. The CycleGAN trained in this way has the ability to transfer style from image domain to image domain. By inputting the source domain image into CycleGAN, a night scene image corresponding to the source domain image can be generated.

[0100] In some embodiments, reference Figure 2 In step S300, the process of performing model training and label generation based on multiple source domain images and multiple target domain images to obtain a first detection model and a target domain image set may include:

[0101] Based on the preset teacher network, multiple target domain images are inferred and processed to obtain several initial candidate boxes for each target domain image;

[0102] According to several initial candidate boxes of each target domain image, a pseudo label of each target domain image is obtained;

[0103] The preset student network is trained according to the training set of the i-th batch; wherein the training set of the i-th batch includes a plurality of target domain images with pseudo labels and source domain images belonging to the i-th batch and with manual labels;

[0104] Update the model parameters of the student network using the supervised loss and unsupervised loss of the student network;

[0105] When i<N, N is a first preset threshold. If the i-th batch does not meet the preset update condition, then set i=i+1 and return to the step of training the preset student network based on the training set of the i-th batch. Otherwise, update the model parameters of the teacher network and return to the step of performing inference processing on multiple target domain images based on the preset teacher network until i≥N. A target domain image set is obtained based on each target domain image and its pseudo label, and the student network after the model parameters are updated is determined as the first detection model.

[0106] The above steps start from i=1.

[0107] In this implementation, a domain adaptation model based on a teacher-student network architecture is proposed for training a cross-domain object detection model. Both the teacher and student networks include an encoder and a prediction head. For ease of understanding, this training process is also referred to as the first stage of training.

[0108] Specifically, first, multiple target domain images are input into a preset teacher network. The teacher network is set to inference mode. The teacher network performs an inference process on the multiple target domain images, and the detection and inference results are saved as several initial candidate boxes for the multiple target domain images. It is worth noting that among the multiple target domain images input into the teacher network, only the first frame target domain image has a manual label, and the other target domain images do not have any labels. In addition, in order to improve the performance of the teacher network, before training, the model parameters obtained based on training multiple source domain images can be used to initialize the teacher network.

[0109] Optionally, the number of initial candidate frames can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the number of initial candidate frames is 25, but is not limited thereto.

[0110] Then, for each target domain image, several initial candidate boxes of the target domain image are screened, aiming to select the highest-quality candidate box from all the initial candidate boxes as the pseudo-label of the target domain image. By traversing all target domain images, pseudo-labels for all target domain images can be obtained.

[0111] Afterwards, multiple source domain images with artificial labels are divided into N batches to obtain source domain images belonging to the N batches. The source domain images belonging to the N batches are integrated with all target domain images with pseudo labels to obtain N batches of training sets. Among them, the training set of the i-th batch can include all target domain images with pseudo labels and source domain images belonging to the i-th batch with artificial labels, i = 1, 2, 3, ..., N. It can be understood that there is at least one source domain image belonging to the i-th batch. After the training set is divided, for the training of the i-th batch, the training set of the i-th batch is input into the student network, that is, the student network simultaneously receives all target domain images with pseudo labels and source domain images belonging to the i-th batch with artificial labels as input, and the two jointly supervise the training process of the student network.

[0112] It is worth noting that in this training process, the student network uses source domain images with manual labels for supervised learning, and uses target domain images with pseudo labels for unsupervised learning.

[0113] After the i-th batch of training is completed, the supervised loss and unsupervised loss of the student network are calculated, and the supervised loss and unsupervised loss of the student network are used to update the model parameters of the student network to obtain the updated model parameters of the student network. It is understood that the model parameters can be set according to actual conditions. For example, the model parameters may include network hyperparameters such as neuron biases and weights, but are not limited to such.

[0114] Finally, determine whether i<N holds.

[0115] If i<N holds, it means that the first stage of training has not yet ended. At this time, it is further judged whether the i-th batch meets the preset update conditions to determine whether the teacher network needs to be updated. If it is determined that the i-th batch meets the update conditions, it means that the teacher network needs to be updated. At this time, the model parameters of the teacher network are updated, and the updated model parameters of the teacher network are obtained as the new model parameters of the teacher network. The process returns to the step of performing inference processing on multiple target domain images based on the preset teacher network to update the pseudo labels generated by the teacher network, thereby improving the quality of the pseudo labels. If it is determined that the i-th batch does not meet the update conditions, it means that the teacher network does not need to be updated for the time being. At this time, the updated model parameters of the student network are used as the new model parameters of the student network, and i=i+1 is set. The process returns to the step of training the preset student network based on the training set of the i-th batch to achieve iterative training.

[0116] If i<N does not hold, it means that the first stage of training can be ended. At this time, each target domain image is used as a sample, and the pseudo label of each target domain image is used as a label. The target domain images and their pseudo labels are integrated into a target domain image set. At the same time, the student network after the model parameters are updated is determined as the first detection model, thereby completing the first stage of training.

[0117] It can be seen that this embodiment provides a domain adaptation model based on the teacher-student architecture, which is used for training cross-domain target detection models. It can not only improve the integrity of the target domain image and reduce the label annotation cost, but also transfer the knowledge of non-extreme scenes to extreme scenes, so that the detection model can fully learn the knowledge of different domains, thereby ensuring that the detection model is adequately trained, reducing the possibility of problems such as overfitting, and improving the robustness of the detection model in extreme scenes.

[0118] Optionally, the encoders and prediction heads configured for the teacher network and the student network can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the teacher network and the student network both use the same Vision Transformer as the encoder and the same Center Head as the prediction head, but are not limited to this.

[0119] Optionally, the first preset threshold can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the first preset threshold can be 250, that is, the first stage training includes 250 batches of training, but is not limited thereto.

[0120] Optionally, the above-mentioned update condition can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above-mentioned update condition can be that the i-th batch is the batch used to update the teacher network. Specifically, during the first stage of training, this embodiment updates the teacher network once every v rounds. The value of v can be set according to actual conditions. For example, v can be 5. Based on this, multiple batches for updating the teacher network can be selected from N batches to obtain a batch set. If the i-th batch is the batch used to update the teacher network, that is, the i-th batch belongs to the batch set, then the i-th batch is determined to meet the update condition; otherwise, the i-th batch is determined to not meet the update condition.

[0121] In some embodiments, obtaining a pseudo label for each target domain image based on the plurality of initial candidate boxes of each target domain image may include:

[0122] Screening several initial candidate frames of the target domain image to obtain multiple target candidate frames of the target domain image;

[0123] Voting is performed on each target candidate box in the target domain image to obtain the voting weight of each target candidate box in the target domain image;

[0124] According to the coordinate position and voting weight of each target candidate box in the target domain image, the pseudo label of the target domain image is obtained.

[0125] In this implementation, the appearance of objects in extreme scenarios often changes dramatically. Currently, pseudo-labeling is commonly used to improve target domain adaptability. However, this approach lacks high-quality filtering of the generated pseudo-labels, resulting in low-quality, noisy pseudo-labels. While a few methods filter generated pseudo-labels, these methods lack optimization and filtering of the model's output, leaving room for improvement in the quality of these pseudo-labels.

[0126] To this end, this embodiment proposes a pseudo-label voting mechanism to improve the quality of pseudo-labels. This pseudo-label mechanism first screens out the top K candidate positions with the highest response values ​​through the classification score heat map, and calculates the confidence of each position based on a soft voting mechanism with Gaussian weights to measure the reliability of the pseudo-label. Positions with higher confidence are then given higher weights, while positions with lower confidence are given lower weights. Finally, the final position of the pseudo-label is adjusted through weighted voting. This method can effectively filter out low-quality pseudo-labels and reduce the positioning error caused by pseudo-label noise, thereby effectively improving the quality of pseudo-labels.

[0127] In the pseudo-label voting mechanism, first, the prediction head of the teacher network performs a convolution operation on the feature information extracted by the encoder of the teacher network to obtain the component features of each initial candidate box of the target domain image. The component features include classification score, scale size and offset. Based on the component features of each initial candidate box, several initial candidate boxes of the target domain image are screened to obtain multiple target candidate boxes of the target domain image. Specifically, based on the classification score, several initial candidate boxes are sorted from large to small and the top K initial candidate boxes are selected as target candidate boxes, as shown in the following formula (3):

[0128]

[0129] In formula (3), S topK represents the set of the first K initial candidate boxes; TopK(·) represents the operation of selecting the first K initial candidate boxes; φ(·) represents the embedding function of the teacher network; Represents a template; represents the search area; θ T Represents the model parameters of the teacher network. It should be noted that in the single target tracking network model, the input image is cropped in two ways according to the initial annotation box. The smaller one is the template, which represents the region of interest, and the larger one is the search area, which represents the region of interest and the Beijing area. The model can accept these two inputs at the same time and find the target similar to the template in the search area and select it as the tracking result, that is, the candidate box.

[0130] Optionally, the number of target candidate frames can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the number of target candidate frames can be 10, but is not limited thereto.

[0131] Then, a pseudo-label voting strategy is performed for the selected K target candidate boxes, which follows the following formula (4):

[0132]

[0133] In formula (4), M p represents the voting weight of the p-th target candidate box; C p represents the classification score of the p-th target candidate box; C min represents the minimum classification score of K target candidate boxes; C max represents the maximum classification score of K target candidate boxes; D(C p , C max ) represents the distance between the classification score of the pth target candidate box and the maximum classification score of the K target candidate boxes; σ p It refers to the standard deviation of the p-th target candidate box obtained by the K-Means clustering algorithm.

[0134] It is worth noting that in the pseudo-label voting strategy, the K-Means clustering algorithm is used to group classification scores into different clusters based on similarity. This similarity refers to the distance between different classification scores or the similarity of features. Specifically, the K-Means clustering algorithm measures similarity by calculating the Euclidean distance between each classification score and the cluster center, and then calculates the standard deviation of each point within the cluster. The clustering algorithm can be used to assign classification scores to similar clusters. The standard deviation of each cluster reflects the degree of dispersion of the points within the cluster, that is, the distance between the points and the cluster center. This standard deviation is used to measure the reliability of each cluster, thereby helping to evaluate the accuracy and stability of pseudo-labeling.

[0135] Finally, a weighted calculation is performed based on the coordinate position and voting weight of each target candidate box in the target domain image to obtain the pseudo label of the target domain image, which is shown in the following formula (5):

[0136]

[0137] In formula (5), B * represents the pseudo label of the target domain image; B p Indicates the coordinate position of the pth target candidate box, which can be expressed as B p ={x1, y1, x2, y2}.

[0138] In some embodiments, the above-mentioned updating of the model parameters of the student network using the supervised loss and unsupervised loss of the student network may include:

[0139] The supervised loss of the student network is obtained based on the predicted labels and manual labels of the source domain images belonging to the i-th batch; the predicted labels of the source domain images are the label information obtained by the student network from predicting the source domain images;

[0140] The unsupervised loss of the student network is obtained based on the predicted labels and pseudo labels of each target domain image. The predicted labels of the target domain images are the label information obtained by the student network for predicting the target domain images.

[0141] According to the supervised loss and unsupervised loss of the student network, the target loss of the student network is obtained;

[0142] Backpropagation is performed using the target loss of the student network to update the model parameters of the student network.

[0143] In this embodiment, the student network is trained using source domain data and target domain data, wherein the student network uses source domain images and their manual labels for supervised learning, and uses target domain images and their pseudo labels for unsupervised learning. Therefore, in the first stage of training, both supervised learning loss function and unsupervised learning loss function are used as the loss function of the student network, so that the effect of consistency loss can be achieved. Through the joint action of the above-mentioned supervised loss and unsupervised loss, the model can match the tracking knowledge of the source domain with the tracking knowledge in extreme scenarios, making the model able to cope with various extreme scenarios and have higher robustness, that is, the model has "consistent" tracking performance in various scenarios.

[0144] For supervised learning loss, the predicted labels of the source domain images belonging to the i-th batch can be obtained through the prediction of the student network. Then, the predicted labels of the source domain images belonging to the i-th batch and the manual labels are used to calculate the classification loss function, regression loss function and intersection-over-union loss function to obtain the supervised loss of the student network, which is shown in the following formula (6):

[0145]

[0146] In formula (6), L track represents the supervision loss of the student network; L cls represents the supervised classification loss of the student network; L iou Represents the supervised IoU loss of the model positioning box of the student network, that is, the supervised intersection-over-union loss; λ iou represents the weight of the supervised intersection-over-union loss; L1 represents the supervised regression loss of the student network, which uses the L1 loss function, namely the mean absolute error (MAE) loss function; represents the weight of the supervised regression loss.

[0147] Optionally, the classification loss function may be set according to actual conditions, and this embodiment does not specifically limit this. For example, the classification loss function may be a cross entropy loss function, but is not limited thereto.

[0148] Optionally, the weight of the supervised regression loss and the weight of the supervised intersection-over-union loss are both set according to actual conditions, and this embodiment does not specifically limit this. For example, the weight of the supervised regression loss may be 5, and the weight of the supervised intersection-over-union loss may be 2, but the present invention is not limited thereto.

[0149] The unsupervised learning loss is roughly the same as the supervised learning loss. The difference is that the unsupervised learning loss uses pseudo labels and their corresponding images. Specifically, the predicted labels of each target domain image can be obtained through the prediction of the student network, and the predicted labels and pseudo labels of each target domain image are used to calculate the classification loss function, regression loss function and intersection-over-union loss function to obtain the unsupervised loss of the student network. Its expression is the same as the above formula (6), which will not be repeated here.

[0150] Afterwards, the supervised loss and unsupervised loss of the student network are weighted to obtain the target loss of the student network, which is shown in the following formula (7):

[0151] L=L track +λL untrack (7);

[0152] In formula (7), L represents the target loss of the student network; L track represents the supervision loss of the student network; L untrack represents the unsupervised loss of the student network; λ represents the weight of the unsupervised loss.

[0153] Optionally, the weight of the unsupervised loss can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the weight of the unsupervised loss can be 0.5, but is not limited thereto.

[0154] Finally, the target loss of the student network is used for backpropagation to update the model parameters of the student network, which can ensure the training accuracy of the student network and thus improve the detection accuracy of the first detection model.

[0155] In some implementations, updating the model parameters of the teacher network may include:

[0156] According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

[0157] In this embodiment, since the teacher network is set to inference mode, the teacher network does not actively update its model parameters. To this end, this embodiment uses the exponential moving average (EMA) method and combines it with the updated model parameters of the student network to update the model parameters of the teacher network to iteratively improve the quality of the pseudo labels it generates. The above process is shown in the following formula (8):

[0158] v t =αv t-1 +(1-α)θ t (8);

[0159] In formula (8), v t represents the updated model parameters of the teacher network in the tth update; v t-1 represents the average value of all updated model parameters of the teacher network in the first t-1 times; θ t represents the updated model parameters of the student network before the tth update; α is the equilibrium constant, which is used to adjust the weight parameter used to average the parameters of the previous number of times. It should be noted that the tth update in this formula refers to the tth update to the teacher network.

[0160] Optionally, the equilibrium constant can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the equilibrium constant can be 0.9, but is not limited thereto.

[0161] In some embodiments, reference Figure 2 In the above step S400, training the first detection model according to the target domain image set to obtain the second detection model may include:

[0162] Obtaining a third detection model according to the first detection model;

[0163] The third detection model is trained according to the j-th batch training set; wherein the j-th batch training set includes target domain images belonging to the j-th batch and having pseudo labels;

[0164] Updating model parameters of the third detection model using the unsupervised loss of the third detection model;

[0165] When j < N', set j = j + 1, and return to the step of training the third detection model according to the j-th batch of training sets until j ≥ N', and determine the third detection model after the model parameters are updated as the second detection model;

[0166] The above steps start from j=1.

[0167] In this embodiment, a third detection model can be obtained based on the first detection model. Exemplarily, the first detection model is directly determined as the third detection model. As another example, some modules for improving the performance of the model are added to the first detection model to obtain a third detection model. These modules can be existing modules, such as modules based on a certain attention mechanism, or non-existing modules, but are not limited to this. It is worth noting that in the embodiment of the present application, a dynamic aggregation prompt module is added to the first detection model to improve the performance of the detection model. Its specific implementation will be explained in the following embodiments. After obtaining the third detection model, the target domain image set obtained in the above steps is used to train the third detection model. This training process can be called the second stage training.

[0168] In this second stage of training, first, the target domain image set is divided into N' batches to obtain N' batches of training sets. Among them, the training set of the jth batch can include target domain images belonging to the jth batch and with pseudo labels, j = 1, 2, 3, ..., N'. It can be understood that there is at least one target domain image belonging to the jth batch and with pseudo labels. Then, for the training of the jth batch, the training set of the jth batch is input into the third detection model, that is, the third detection model receives the target domain images belonging to the jth batch and with pseudo labels and is trained based on them to learn the relationship between the target domain images and their pseudo labels. After the training of the jth batch is completed, the unsupervised loss of the third detection model is calculated, and the unsupervised loss of the third detection model is used for backpropagation to update the model parameters of the third detection model to obtain the updated model parameters of the third detection model. It can be understood that the specific implementation of calculating the unsupervised loss of the third detection model is the same as the specific implementation of calculating the unsupervised loss of the student network in the aforementioned embodiment, and will not be repeated here. Finally, it is determined whether j < N'. If j < N' holds, it indicates that the second phase of training has not yet concluded. The updated model parameters of the third detection model are then determined as the new model parameters of the third detection model. Set j = j + 1, and return to the step of training the third detection model based on the jth batch of training data to implement iterative training. If j < N' does not hold, it indicates that the second phase of training can be concluded. The third detection model with updated model parameters is then determined as the second detection model.

[0169] It can be seen that in this embodiment, the first detection model is trained using target domain images with pseudo labels, so that the detection model can fully learn the mapping relationship between the target domain images and their labels, thereby having the ability to perform target detection tasks in extreme scenarios, which can effectively improve the performance of the detection model in target detection tasks in extreme scenarios.

[0170] Optionally, the second preset threshold can be flexibly set according to actual conditions, and this embodiment does not specifically limit this. For example, the second preset threshold is 50, that is, the second stage training includes 50 batches of training, but is not limited to this.

[0171] In some embodiments, reference Figure 3 , the second detection model may include an input layer, a feature extraction structure, and a prediction head, wherein:

[0172] The input layer is used to process the image to be tested into blocks to obtain an image block sequence;

[0173] The feature extraction structure includes multiple feature extraction modules connected in sequence. The input features of the first feature extraction module are the image block sequence. The input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module. The feature extraction module includes an encoder and a dynamic aggregation prompt module. The encoder is used to extract features from the input features of the feature extraction module to obtain the search features of the feature extraction module. The dynamic aggregation prompt module is used to obtain the output features of the feature extraction module based on the search features and input features of the feature extraction module.

[0174] The prediction head is used to make predictions based on the output features of the last feature extraction module to obtain the target detection results of the image to be tested.

[0175] In this embodiment, it can be seen from the above embodiments that the first detection model is a trained student network, and in some examples, the student network can use Vision-Transformer as an encoder and Center Head as a prediction head. Based on this, the first detection model of this embodiment can use Vision-Transformer as the basic network and Center Head as the prediction head. In order to improve the performance of the detection model, a dynamic aggregation prompt module is added between each layer of Transformer encoder of Vision-Transformer, thereby obtaining a third detection model. The second detection model can be obtained by training the third detection model.

[0176] In real-time target detection tasks, the image to be tested is input into the second detection model. In the second detection model, first, the input layer divides the image to be tested into blocks and adds position encoding to obtain an image block sequence, which is input into the feature extraction structure; then, the feature extraction structure uses multiple feature extraction modules to perform multiple rounds of feature extraction on the image block sequence; finally, the Center Head prediction head predicts the output features of the last feature extraction module to obtain the target detection result of the image to be tested, thus achieving target detection tasks in extreme scenarios.

[0177] Specifically, the above-mentioned feature extraction structure may include a plurality of feature extraction modules connected in sequence, the input features of the first feature extraction module are image block sequences, and the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module. Each feature extraction module includes a Transformer encoder and a dynamic aggregation prompt module. In each feature extraction module, the input features of the feature extraction module are subjected to feature extraction by the Transformer encoder to obtain the search features of the feature extraction module, and then the search features and input features of the feature extraction module are aggregated using the dynamic aggregation prompt module to obtain the output features of the feature extraction module. This can improve the feature extraction capability of the second detection model in extreme scenarios, thereby improving the target detection accuracy of the second detection model in extreme scenarios.

[0178] Optionally, the number of the feature extraction modules can be set according to actual conditions, and this embodiment does not impose any specific limitation on this.

[0179] In some embodiments, in the dynamic aggregation prompt module, the process of obtaining the output features of the feature extraction module based on the search features and input features of the feature extraction module may include:

[0180] According to the search features and input features of the feature extraction module, the fusion features of the feature extraction module are obtained;

[0181] According to the search features and input features of the feature extraction module, dynamic prompts of the feature extraction module are obtained;

[0182] According to the dynamic prompts and search features of the feature extraction module, the output features of the feature extraction module are obtained.

[0183] In this embodiment, the dynamic aggregation prompt module is intended to enhance the detection model's adaptability to changes in target appearance through a dynamic prompt mechanism. It first fuses the search features of the current stage with the updated search features of the previous stage and balances their weights to generate hierarchical visual features, which can better capture the contextual feature information of the target. Then, the global context feature information of the search features of the current stage and the updated search features of the previous stage is extracted, and these global context feature information are encoded to generate dynamic prompts with scene adaptability. Finally, the dynamic prompt is used to update the search features of the current stage to obtain the final output features, which can further enhance the model's ability to model changes in target appearance and its robustness to extreme scene interference.

[0184] Specifically, in order to fuse the layered visual semantic information and obtain hierarchical visual features, this embodiment first uses the neighbor search features for fusion. Specifically, based on the search features and input features of the feature extraction module, the fusion features of the feature extraction module are obtained, as shown in the following formula (9):

[0185] X′ l =G(X l ;θ)⊙X l +(1-G(X l-1 ;θ))⊙X l-1 (9);

[0186] In formula (9), X′ l represents the fusion feature of the lth feature extraction module; X l represents the search feature of the lth feature extraction module; X l-1 represents the input features of the lth feature extraction module, i.e., the output features of the l-1th feature extraction module, which can be understood as the updated search features of the l-1th feature extraction module; G(·) represents a sigmoid gating function with bilinear upsampling, which is used to adjust the feature fusion weights; θ represents a preset learnable parameter that can be flexibly set based on actual conditions; ⊙ represents element-wise multiplication. This process extracts hierarchical visual semantic information by layering the current search features with the updated search features of the previous layer, thus helping the model adapt to changes in the appearance of the target object.

[0187] At the same time, in order to enhance the model's adaptability to changes in target appearance, this embodiment proposes a dynamic prompt for the aggregation of search features. Specifically, based on the search features and input features of the feature extraction module, a dynamic prompt of the feature extraction module is obtained, as shown in the following formula (10):

[0188]

[0189] In formula (10), P l Represents the dynamic prompt of the lth feature extraction module; Represents X l The u-th element in ; Represents X l-1 The u-th element in ; Represents preset learnable parameters, which can be flexibly set according to actual conditions; Represents the sequentially connected convolutional and pooling layers. The pooling operation type of the pooling layer can be flexibly set according to the actual situation. For example, the pooling operation can be global average pooling, but it is not limited to this. This process uses the attention mechanism to assign corresponding weights to each search feature. Dynamic cues enable the model to effectively capture the changing patterns of the target appearance.

[0190] Finally, the generated dynamic prompts are used to adjust the target features extracted by the model by multiplying the corresponding elements. Specifically, the dynamic prompts of the feature extraction module are used to update the search features of the feature extraction module, thereby obtaining the updated search features of the feature extraction module as the output features of the feature extraction module, as shown in the following formula (11):

[0191] X l =X l ·P l (11).

[0192] To facilitate the understanding of the domain adaptive target detection method in the above-mentioned extreme scenarios in the embodiment of the present application, the domain adaptive target detection method in the above-mentioned extreme scenarios in the embodiment of the present application will be described below using an application scenario. In this application scenario, the target domain is a foggy scene and a night scene (i.e., an extreme scene), and the source domain is a daytime scene (i.e., a non-extreme scene). The domain adaptive target detection method in the above-mentioned extreme scenes can be divided into four parts: image synthesis, first-stage training, second-stage training, and real-time target detection, which are specifically implemented as follows:

[0193] S01, image synthesis:

[0194] First, multiple daytime scene images are obtained as source domain images. For each daytime scene image, the daytime scene image is first input into the ViT-based DINOv2 encoder of depthanything to extract high-dimensional features. The high-dimensional features are then passed to the DPT-based decoder of depthanything. The decoder maps the features into a depth prediction map, i.e., depth information. Then, the daytime scene image and its depth information are combined with the fog scattering model shown in the above formulas (1)-(2) to obtain the foggy scene image corresponding to the daytime scene image. At the same time, the daytime scene image is input into the trained CycleGAN to generate the nighttime scene image corresponding to the source domain image. Then, only the manual label of the first daytime scene image is assigned to the foggy scene image and nighttime scene image corresponding to the first daytime scene image, while the foggy scene images and nighttime scene images corresponding to the other daytime scene images do not have any labels. In this way, multiple foggy scene images and multiple nighttime scene images can be obtained as target domain images.

[0195] S02, the first stage of training, which includes 250 batches of training:

[0196] First, a teacher network and a student network are constructed. Both the teacher network and the student network use the same VisionTransformer as the encoder and the same Center Head as the prediction head. Before training, the teacher network is initialized using the model parameters obtained by training based on multiple source domain images.

[0197] Then, before the i-th batch training, the teacher network is used to perform an inference process on multiple target domain images, and the detection and inference results are saved as 25 initial candidate frames of multiple target domain images. For each target domain image, the 25 initial candidate frames are first sorted from large to small based on their classification scores, and the top 10 initial candidate frames are selected as target candidate frames, which is implemented as shown in the above formula (3). Then, a pseudo-label voting strategy is performed for the selected 10 target candidate frames to obtain the voting weights of these 10 target candidate frames, which follows the above formula (4). Finally, a weighted calculation is performed based on the coordinate positions and voting weights of these 10 target candidate frames to obtain the pseudo-label of the target domain image, which is implemented as shown in the above formula (5). By traversing all target domain images, the pseudo-labels of all target domain images can be obtained.

[0198] Afterwards, during the i-th batch training, the training set of the i-th batch is input into the student network. That is, the student network simultaneously receives all pseudo-labeled target domain images and the manually labeled source domain images belonging to the i-th batch as input, and the two jointly supervise the training process of the student network. After the i-th batch training is completed, the target loss of the student network is calculated. The target loss is composed of the unsupervised loss and the supervised loss, as shown in the above formulas (6)-(7). The target loss of the student network is used for backpropagation to update the model parameters of the student network.

[0199] Finally, determine whether i<N holds true, where N is 250.

[0200] If i<N holds true, it is further determined whether the i-th batch is the batch used to update the teacher network.

[0201] If the i-th batch is determined to be the batch for updating the teacher network, the model parameters of the teacher network are updated based on the updated model parameters of the student network combined with the exponential moving average method. This is implemented as shown in the above formula (8). The updated model parameters of the teacher network are obtained as the new model parameters of the teacher network, and the process returns to the step of performing inference on multiple target domain images through the teacher network to update the pseudo labels generated by the teacher network, thereby improving the quality of the pseudo labels. In this application scenario, the teacher network is updated once every 5 rounds, that is, before the 1st round of training, the 6th round of training, the 11th round of training, ..., and the 246th round of training.

[0202] If it is determined that the i-th batch is not the batch used to update the teacher network, the updated model parameters of the student network are used as the new model parameters of the student network, i=i+1 is set, and the training step of the i-th batch is returned to realize iterative training.

[0203] If i<N does not hold, then each target domain image is used as a sample, and the pseudo label of each target domain image is used as a label. The target domain images and their pseudo labels are integrated into a target domain image set. At the same time, the student network with updated model parameters is determined as the first detection model, thereby completing the first stage of training.

[0204] S03, the second stage of training, which includes 50 batches:

[0205] First, the first detection model uses Vision-Transformer as the basic network and Center Head as the prediction head. In order to improve the performance of the detection model, a dynamic aggregation prompt module is added between each layer of Transformer encoder of Vision-Transformer to obtain the third detection model.

[0206] Then, the target domain image set is divided into N' batches to obtain N' batches of training sets, wherein the j-th batch of training sets may include target domain images belonging to the j-th batch and having pseudo labels.

[0207] Subsequently, during the j-th batch of training, the j-th batch of training data is input into the third detection model. Specifically, the third detection model receives pseudo-labeled target domain images from the j-th batch and is trained based on these target domain images to learn the relationship between the target domain images and their pseudo-labels. After the j-th batch of training is complete, the unsupervised loss of the third detection model is calculated, and backpropagation is performed using the unsupervised loss of the third detection model to update the model parameters of the third detection model, obtaining the updated model parameters of the third detection model.

[0208] Finally, determine whether j < N' holds, where N' is 50. If j < N' holds, the updated model parameters of the third detection model are determined as the new model parameters of the third detection model, j = j + 1, and the process returns to the step of training the third detection model based on the j-th batch of training data to implement iterative training. If j < N' does not hold, the third detection model with updated model parameters is determined as the second detection model.

[0209] S04, real-time target detection:

[0210] First, the input layer divides the image under test into blocks and adds positional encodings to generate a sequence of image blocks. This sequence of image blocks is then input into the feature extraction structure. The feature extraction structure then uses multiple feature extraction modules to perform multiple rounds of feature extraction on the image block sequence. Finally, the Center Head predicts the output features of the last feature extraction module to obtain the object detection result, such as a vehicle, in the image under test.

[0211] Specifically, the feature extraction structure includes multiple feature extraction modules connected in sequence. The input features of the first feature extraction module are image block sequences, and the input features of other feature extraction modules except the first feature extraction module are the output features of the previous feature extraction module. Each feature extraction module includes a Transformer encoder and a dynamic aggregation prompt module. In each feature extraction module, the input features of the feature extraction module are first extracted through the Transformer encoder to obtain the search features of the feature extraction module. Then, the search features of the feature extraction module and the input features are aggregated using the dynamic aggregation prompt module to obtain the output features of the feature extraction module.

[0212] More specifically, in the dynamic aggregation prompt module, first, based on the search features and input features of the feature extraction module, the fusion features of the feature extraction module are obtained, as shown in the above formula (9); then, based on the search features and input features of the feature extraction module, the dynamic prompts of the feature extraction module are obtained, as shown in the above formula (10); finally, based on the dynamic prompts and search features of the feature extraction module, the output features of the feature extraction module are obtained, as shown in the above formula (11).

[0213] In order to further verify the effect of the embodiment of the present application, the embodiment of the present application was verified on the dataset Foggy-Cityscapes and the dataset BDD100K. After verification, the embodiment of the present application demonstrated excellent performance in extreme scenarios of foggy days and nighttime, reaching the current SOTA level. Specifically, the model effectively alleviates the semantic offset problem between the source domain and the target domain by optimizing the pseudo-label quality and feature alignment capabilities, and exhibits excellent accuracy and robustness in a variety of complex scenarios. At the same time, the dynamic aggregation prompt module enhances the model's adaptability to drastic changes in target appearance, enabling it to maintain stable target detection performance in an interfering background. The mAP of the embodiment of the present application on the dataset Foggy-Cityscapes reaches 60.5%, and the detection performance of targets in low-quality images on the dataset BDD100K reaches 63.1%. In addition, the model has extremely high real-time detection performance, running at a speed of 110FPS on GPU devices and 38FPS on embedded devices. It can not only accurately complete target positioning tasks, but also meet the real-time requirements of target detection in actual complex environments, fully verifying the practicality and reliability of the model.

[0214] In addition, refer to Figure 3 , the present application also provides a domain adaptive target detection device for extreme scenarios, which may include:

[0215] An acquisition module 101 is configured to acquire a plurality of source domain images with artificial labels, where the source domain images are non-extreme scene images;

[0216] A first processing module 102 is configured to obtain multiple target domain images based on multiple source domain images, where the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels;

[0217] A second processing module 103 is configured to perform model training and label generation processing based on the multiple source domain images and the multiple target domain images to obtain a first detection model and a target domain image set, where the target domain image set includes multiple target domain images with pseudo labels;

[0218] The third processing module 104 is configured to train the first detection model based on the target domain image set to obtain a second detection model;

[0219] The fourth processing module 105 is configured to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0220] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0221] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0222] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A domain adaptive target detection method in extreme scenarios, characterized by: The following steps are involved: Acquire a plurality of source domain images with manual labels; wherein the source domain images are non-extreme scene images; A plurality of target domain images are obtained based on the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels; Performing model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes the plurality of target domain images with pseudo labels; Training the first detection model according to the target domain image set to obtain a second detection model; Performing target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested; The second detection model includes a feature extraction structure, the feature extraction structure includes a plurality of sequentially connected feature extraction modules, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to extract features from input features of the feature extraction module to obtain search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain output features of the feature extraction module based on the search features and the input features of the feature extraction module; In the dynamic aggregation prompt module, the search features and the input features of the feature extraction module are fused and weighted to obtain the fused features of the feature extraction module; the dynamic prompt of the feature extraction module is obtained based on the search features and the input features of the feature extraction module; the dynamic prompt of the feature extraction module is used to update the fused features of the feature extraction module by multiplying the corresponding elements to obtain the output features of the feature extraction module; wherein the dynamic prompt satisfies the following formula: ； Where, Indicates the Dynamic prompts of feature extraction modules; express The elements, Indicates the Search features of feature extraction modules; express The elements, Indicates the Input features of feature extraction modules; Represents preset learnable parameters; Represents the convolutional layers and pooling layers connected sequentially; Represents element-wise multiplication.

2. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The step of obtaining a plurality of target domain images according to the plurality of source domain images comprises: The image generation processing of the extreme scene is performed based on each of the source domain images to obtain the target domain image corresponding to each of the source domain images, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining a plurality of target domain images.

3. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The performing model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set includes: Performing inference processing on the plurality of target domain images based on a preset teacher network to obtain a plurality of initial candidate frames for each of the target domain images; Obtaining the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image; According to i The batch training set is used to train the preset student network; wherein, the first i The training set of a batch includes a plurality of target domain images with the pseudo labels and images belonging to the first i batches of source domain images with the manual labels; Updating the model parameters of the student network using the supervised loss and unsupervised loss of the student network; when i < N hour, N is the first preset threshold, if the i If the batch does not meet the preset update conditions, i=i+ 1, and return to the i The step of training the preset student network based on the batch training set, otherwise updating the model parameters of the teacher network, and returning to the step of performing inference processing on the plurality of target domain images based on the preset teacher network until i ≥ N , obtaining the target domain image set according to each of the target domain images and the pseudo labels thereof, and determining the student network after the model parameters are updated as the first detection model; Among them, the above steps are i =1 to start.

4. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The obtaining of the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image comprises: Performing screening processing on the plurality of initial candidate frames of the target domain image to obtain a plurality of target candidate frames of the target domain image; Performing voting on each of the target candidate boxes in the target domain image to obtain a voting weight for each of the target candidate boxes in the target domain image; The pseudo label of the target domain image is obtained according to the coordinate position and voting weight of each target candidate box of the target domain image.

5. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The method of updating the model parameters of the student network using the supervised loss and the unsupervised loss of the student network comprises: According to the i The predicted labels of the source domain images of the batch and the manual labels are used to obtain the supervision loss of the student network; wherein the predicted labels of the source domain images are label information obtained by the student network predicting the source domain images; Obtaining an unsupervised loss of the student network based on the predicted label of each target domain image and the pseudo label; wherein the predicted label of the target domain image is label information obtained by predicting the target domain image by the student network; Obtaining a target loss of the student network according to the supervised loss and the unsupervised loss of the student network; Backpropagation is performed using the target loss of the student network to update the model parameters of the student network.

6. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The updating of the model parameters of the teacher network comprises: According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

7. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The training of the first detection model according to the target domain image set to obtain a second detection model includes: Obtaining a third detection model according to the first detection model; According to j The third detection model is trained using a batch of training sets; wherein the first j The training set of the batch includes j a batch of the target domain images with the pseudo labels; Updating model parameters of the third detection model using the unsupervised loss of the third detection model; when j < N’ hour, N’ is the second preset threshold, let j=j+ 1, and return to the j The step of training the third detection model using the batch training set until j ≥ N’ , determining the third detection model after the model parameters are updated as the second detection model; Among them, the above steps are j =1 to start.

8. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The second detection model also includes an input layer and a prediction head. The input layer is used to perform block processing on the image to be tested to obtain an image block sequence. The input features of the first feature extraction module are the image block sequence, and the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module. The prediction head is used to make predictions based on the output features of the last feature extraction module to obtain the target detection result of the image to be tested.

9. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: Obtaining the output features of the feature extraction module according to the search features and the input features of the feature extraction module includes: Obtaining the fusion feature of the feature extraction module according to the search feature and the input feature of the feature extraction module; Obtaining the dynamic prompt of the feature extraction module according to the search feature and the input feature of the feature extraction module; The output features of the feature extraction module are obtained according to the dynamic prompts and the fusion features of the feature extraction module.

10. A domain adaptive target detection device for extreme scenarios, characterized in that: include: An acquisition module, configured to acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images; A first processing module is configured to obtain a plurality of target domain images based on the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels; a second processing module, configured to perform model training and label generation processing based on the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels; a third processing module, configured to train the first detection model according to the target domain image set to obtain a second detection model; A fourth processing module is configured to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested; The second detection model includes a feature extraction structure, the feature extraction structure includes a plurality of sequentially connected feature extraction modules, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to extract features from input features of the feature extraction module to obtain search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain output features of the feature extraction module based on the search features and the input features of the feature extraction module; In the dynamic aggregation prompt module, the search features and the input features of the feature extraction module are fused and weighted to obtain the fused features of the feature extraction module; the dynamic prompt of the feature extraction module is obtained based on the search features and the input features of the feature extraction module; the dynamic prompt of the feature extraction module is used to update the fused features of the feature extraction module by multiplying the corresponding elements to obtain the output features of the feature extraction module; wherein the dynamic prompt satisfies the following formula: ； Where, Indicates the Dynamic prompts for feature extraction modules; express The elements, Indicates the Search features of feature extraction modules; express The elements, Indicates the Input features of feature extraction modules; Represents preset learnable parameters; Represents the convolutional layers and pooling layers connected sequentially; Represents element-wise multiplication.

Images