Domain adaptive target detection method and device in extreme scene

Abstract

The invention discloses a domain adaptive target detection method and device in an extreme scene, and the method comprises the steps: obtaining a plurality of source domain images with artificial labels, the source domain images being non-extreme scene images; according to the plurality of source domain images, a plurality of target domain images are obtained, the target domain images are extreme scene images, only the first target domain image has an artificial label, and the other target domain images have no labels; performing model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set, the target domain image set comprising a plurality of target domain images with pseudo labels; training the first detection model according to the target domain image set to obtain a second detection model; and performing target detection on the to-be-detected image according to the second detection model to obtain a target detection result of the to-be-detected image. The method can effectively improve the target detection precision for extreme scenes, and can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a domain adaptive target detection method and device in extreme scenarios. Background Art

[0002] The object detection task aims to automatically identify and locate a specific type of object from a given image or video frame. Currently, extreme scene images with artificial labels are usually used to train the object detection model, and the trained object detection model is used to achieve the object detection task in extreme scenes. However, since extreme scene images are often difficult to obtain, the number of extreme scene image samples is limited, which leads to insufficient training of the object detection model, which in turn affects the performance of the object detection model in the object detection task in extreme scenes. Its object detection accuracy for extreme scenes needs to be improved. Summary of the invention

[0003] The embodiments of the present application provide a domain adaptive target detection method and device in extreme scenarios, which are used to improve the target detection accuracy in extreme scenarios.

[0004] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0005] On the one hand, an embodiment of the present application provides a domain adaptive target detection method in extreme scenarios, comprising the following steps:

[0006] Acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images;

[0007] According to the plurality of source domain images, a plurality of target domain images are obtained; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels;

[0008] Performing model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels;

[0009] Training the first detection model according to the target domain image set to obtain a second detection model;

[0010] Perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0011] Furthermore, in one embodiment, obtaining a plurality of target domain images according to the plurality of source domain images comprises:

[0012] The image generation process of the extreme scene is performed based on each of the source domain images to obtain the target domain image corresponding to each of the source domain images, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining a plurality of the target domain images.

[0013] Furthermore, in one embodiment, the performing model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set includes:

[0014] Performing reasoning processing on the plurality of target domain images based on a preset teacher network to obtain a plurality of initial candidate frames of each of the target domain images;

[0015] Obtaining the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image;

[0016] The preset student network is trained according to the i-th batch of training sets; wherein the i-th batch of training sets includes a plurality of the target domain images with the pseudo labels and the source domain images belonging to the i-th batch and having the artificial labels;

[0017] Updating the model parameters of the student network using the supervised loss and the unsupervised loss of the student network;

[0018] When i<N, N is a first preset threshold value, if the i-th batch does not meet the preset update condition, then set i=i+1, and return to the step of training the preset student network according to the training set of the i-th batch, otherwise update the model parameters of the teacher network, and return to the step of performing inference processing on the plurality of target domain images based on the preset teacher network, until i≥N, obtain the target domain image set according to each of the target domain images and their pseudo labels, and determine the student network after the model parameters are updated as the first detection model;

[0019] The above steps start from i=1.

[0020] Furthermore, in one embodiment, obtaining the pseudo label of each target domain image according to the plurality of initial candidate boxes of each target domain image includes:

[0021] Screening the plurality of initial candidate frames of the target domain image to obtain a plurality of target candidate frames of the target domain image;

[0022] Performing voting processing on each of the target candidate boxes in the target domain image to obtain a voting weight of each of the target candidate boxes in the target domain image;

[0023] The pseudo label of the target domain image is obtained according to the coordinate position and voting weight of each target candidate box of the target domain image.

[0024] Furthermore, in one embodiment, the updating of the model parameters of the student network using the supervised loss and the unsupervised loss of the student network comprises:

[0025] Obtaining the supervision loss of the student network according to the predicted labels of the source domain images belonging to the i-th batch and the manual labels; wherein the predicted labels of the source domain images are label information obtained by predicting the source domain images by the student network;

[0026] According to the predicted labels of each of the target domain images and the pseudo labels, an unsupervised loss of the student network is obtained; wherein the predicted labels of the target domain images are label information obtained by predicting the target domain images by the student network;

[0027] Obtaining a target loss of the student network according to the supervised loss and the unsupervised loss of the student network;

[0028] Back-propagation is performed using the target loss of the student network to update the model parameters of the student network.

[0029] Further, in one embodiment, updating the model parameters of the teacher network includes:

[0030] According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

[0031] Furthermore, in one embodiment, the training of the first detection model according to the target domain image set to obtain the second detection model includes:

[0032] According to the first detection model, a third detection model is obtained;

[0033] The third detection model is trained according to the j-th batch of training sets; wherein the j-th batch of training sets includes the target domain images belonging to the j-th batch and having the pseudo labels;

[0034] Updating model parameters of the third detection model using the unsupervised loss of the third detection model;

[0035] When j<N', N' is a second preset threshold, let j=j+1, and return to the step of training the third detection model according to the jth batch of training sets until j≥N', and determine the third detection model after the model parameters are updated as the second detection model;

[0036] The above steps start from j=1.

[0037] Further, in one embodiment, the second detection model includes an input layer, a feature extraction structure and a prediction head, wherein:

[0038] The input layer is used to process the image to be tested in blocks to obtain an image block sequence;

[0039] The feature extraction structure includes a plurality of feature extraction modules connected in sequence, the input feature of the first feature extraction module is the image block sequence, the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to perform feature extraction on the input features of the feature extraction module to obtain the search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain the output features of the feature extraction module according to the search features and the input features of the feature extraction module;

[0040] The prediction head is used to make predictions based on the output features of the last feature extraction module to obtain the target detection result of the image to be tested.

[0041] Further, in one embodiment, obtaining the output feature of the feature extraction module based on the search feature and the input feature of the feature extraction module comprises:

[0042] Obtaining a fusion feature of the feature extraction module according to the search feature and the input feature of the feature extraction module;

[0043] Obtaining a dynamic prompt of the feature extraction module according to the search feature and the input feature of the feature extraction module;

[0044] The output feature of the feature extraction module is obtained according to the dynamic prompt and the search feature of the feature extraction module.

[0045] On the other hand, an embodiment of the present application provides a domain adaptive target detection device in extreme scenarios, including:

[0046] An acquisition module, used to acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images;

[0047] A first processing module is used to obtain a plurality of target domain images according to the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels;

[0048] A second processing module is used to perform model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels;

[0049] A third processing module, configured to train the first detection model according to the target domain image set to obtain a second detection model;

[0050] The fourth processing module is used to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0051] According to the domain adaptive target detection method and device in extreme scenarios of the embodiments of the present application, first, multiple target domain images are generated by using multiple source domain images with artificial labels, wherein the source domain images are non-extreme scene images, and the target domain images are extreme scene images, and only the first target domain image has an artificial label, and the remaining target domain images are unlabeled, so that the number of samples of target domain images can be effectively increased and the diversity of target domain images can be improved; then, model training and label generation processing are performed based on the multiple source domain images and the multiple target domain images, aiming to generate a corresponding pseudo label for each target domain image, and supervised learning and unsupervised learning are performed on the detection model used to perform target detection tasks in extreme scenarios, so as to obtain a first detection model and A target domain image set, wherein the target domain image set includes multiple target domain images with pseudo labels, which can not only improve the integrity of the target domain images and reduce the label annotation cost, but also ensure that the detection model is adequately trained, reduce the possibility of problems such as overfitting, and improve the robustness of the detection model; then, use the target domain image set to train the first detection model, so that the detection model further learns the relationship between the target domain image and its pseudo label, and then obtains the second detection model; finally, according to the second detection model, target detection is performed on the image to be tested to obtain the target detection result of the image to be tested, thereby realizing the target detection task of extreme scenes. In this way, the embodiment of the present application can effectively improve the target detection accuracy for extreme scenes.

[0052] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of the domain adaptive target detection method in extreme scenarios provided by this application;

[0054] Figure 2 It is a schematic diagram of the domain adaptive target detection method in extreme scenarios provided by this application;

[0055] Figure 3 It is a structural diagram of a domain adaptive target detection device for extreme scenarios provided in this application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0058] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] The object detection task aims to automatically identify and locate a specific type of object from a given image or video frame. Currently, extreme scene images with artificial labels are usually used to train the object detection model, and the trained object detection model is used to achieve the object detection task in extreme scenes. However, since extreme scene images are often difficult to obtain, the number of extreme scene image samples is limited, which leads to insufficient training of the object detection model, which in turn affects the performance of the object detection model in the object detection task in extreme scenes. Its object detection accuracy for extreme scenes needs to be improved.

[0061] In view of this, an embodiment of the present application provides a domain adaptive target detection method and device in extreme scenarios, aiming to improve the target detection accuracy for extreme scenarios.

[0062] First, the implementation steps of the domain adaptive target detection method in extreme scenarios provided by the present application will be described in detail below with reference to the accompanying drawings.

[0063] The domain adaptive target detection method in extreme scenarios provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0064] Reference Figure 1 , Figure 1 It is a flowchart of a domain adaptive target detection method in an extreme scenario provided by the present application. The domain adaptive target detection method may include the following steps S100-S500.

[0065] S100, obtaining multiple source domain images with manual labels.

[0066] It should be noted that manual labels refer to label information obtained by manually annotating images. The label information can be flexibly set according to actual conditions. For example, the label information may include at least one of the location or type of the area of ​​interest in the image, but is not limited to this.

[0067] It is understandable that the source domain refers to a non-extreme scene, and further, the source domain image refers to a non-extreme scene image, that is, an image associated with a non-extreme scene. It is worth noting that the non-extreme scene in the embodiment of the present application is a daytime scene, that is, the source domain image refers to a daytime scene image.

[0068] In this step, first, a plurality of initial source domain images are obtained; then, corresponding label information is annotated for each source domain image by manual labeling, so that each source domain image has a corresponding manual label, thereby obtaining a plurality of source domain images with manual labels.

[0069] S200, obtaining multiple target domain images according to multiple source domain images.

[0070] It should be noted that only the first target domain image has a manual label, and the remaining target domain images are unlabeled.

[0071] It is understandable that the target domain refers to extreme scenes, and further, the target domain image refers to extreme scene images, that is, images associated with extreme scenes. It is worth noting that the extreme scenes in the embodiment of the present application are foggy scenes and night scenes, and further, the target domain image includes night scene images and foggy scene images.

[0072] In this step, after obtaining multiple source domain images, image generation processing is performed on the multiple source domain images so that each source domain image is converted into a corresponding target domain image, thereby obtaining multiple target domain images, which can effectively increase the number of samples of target domain images and improve the diversity of target domain images. Among them, only the first target domain image has an artificial label, and the remaining target domain images have no labels.

[0073] S300, performing model training and label generation processing according to multiple source domain images and multiple target domain images to obtain a first detection model and a target domain image set.

[0074] It should be noted that the target domain image set may include multiple target domain images with pseudo labels, where the pseudo label refers to label information obtained by predicting the target domain image through a certain network.

[0075] In this step, cross-domain model training and label generation processing are performed based on multiple source domain images and multiple target domain images. The implementation of this processing mainly depends on a preset teacher network and a preset student network, wherein the teacher network is used to generate corresponding pseudo labels for each target domain image and regularly update the generated pseudo labels, while the student network is used to iteratively train based on source domain images with artificial labels and target domain images with pseudo labels and regularly update its own model parameters and the model parameters of the teacher network. Through the synergy of the teacher network and the student network, cross-domain knowledge transfer and pseudo label generation of target domain images can be achieved. The first detection model and the target domain image set are obtained through the above processing, wherein the first detection model is the iteratively trained student network, and the target domain image set includes multiple target domain images with pseudo labels generated by the teacher network.

[0076] S400, training the first detection model according to the target domain image set to obtain a second detection model.

[0077] It should be noted that the second detection model refers to a model suitable for target detection tasks in extreme scenarios.

[0078] In this step, after obtaining a plurality of target domain images with pseudo labels, the first detection model is trained using these target domain images with pseudo labels so that the detection model has the ability to perform target detection tasks in extreme scenarios, thereby obtaining a second detection model.

[0079] S500, performing target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0080] It should be noted that the image to be tested refers to an image to be tested and associated with an extreme scene, such as a night scene image to be tested or a foggy scene image to be tested.

[0081] It can be understood that the target detection result refers to the result obtained by performing target detection on the image to be tested, and its type can be flexibly set according to actual conditions. For example, the target detection result may include at least one of the location or type of the area of ​​interest of the image to be tested, but is not limited to this.

[0082] In this step, the image to be tested is input into the second detection model, and prediction processing is performed by the second detection model to obtain the target detection result of the image to be tested, thereby achieving the target detection task in extreme scenes.

[0083] It can be seen that in the embodiments of the present application, first, multiple target domain images are generated by using multiple source domain images with artificial labels, the source domain images are non-extreme scene images, and the target domain images are extreme scene images, and only the first target domain image has an artificial label, and the remaining target domain images are unlabeled, which can effectively increase the number of target domain image samples and improve the diversity of target domain images; then, model training and label generation processing are performed based on the multiple source domain images and the multiple target domain images, aiming to generate corresponding pseudo labels for each target domain image, and supervised learning and unsupervised learning are performed on the detection model used to perform target detection tasks in extreme scenes, thereby obtaining A first detection model and a target domain image set, wherein the target domain image set includes a plurality of target domain images with pseudo labels, which can not only improve the integrity of the target domain images and reduce the label annotation cost, but also ensure that the detection model is adequately trained, reduce the possibility of problems such as overfitting, and improve the robustness of the detection model; then, the first detection model is trained using the target domain image set so that the detection model further learns the relationship between the target domain image and its pseudo label, thereby obtaining a second detection model; finally, target detection is performed on the image to be tested according to the second detection model to obtain the target detection result of the image to be tested, thereby achieving the target detection task of extreme scenes. In this way, the embodiment of the present application can effectively improve the accuracy of target detection for extreme scenes.

[0084] The above steps will be further described below.

[0085] In some embodiments, reference Figure 2 In the above step S200, the implementation process of obtaining multiple target domain images according to multiple source domain images may include:

[0086] Based on each source domain image, image generation processing of extreme scenes is performed to obtain target domain images corresponding to each source domain image, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining multiple target domain images.

[0087] In this embodiment, first, for each source domain image, an extreme scene image generation process is performed based on the source domain image to obtain a target domain image corresponding to the source domain image. By traversing all source domain images, target domain images corresponding to all source domain images can be obtained. It can be understood that the target domain image refers to an extreme scene image, which may include night scene images and foggy scene images. Then, only the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, while the target domain images corresponding to other source domain images do not have any labels. In this way, multiple target domain images can be obtained.

[0088] Specifically, the above image generation process is divided into two parts: foggy scene image generation and night scene image generation.

[0089] In terms of foggy scene image generation, a depth estimation model is used to obtain the depth information of the source domain image. Specifically, the source domain image is input into the depth estimation model, the high-dimensional features of the source domain image are extracted through the encoder of the depth estimation model, and the high-dimensional features are mapped to a depth prediction map as the depth information through the decoder of the depth estimation model. After obtaining the depth information of the source domain image, the source domain image and its depth information are synthesized into a foggy image based on a preset fog scattering model to obtain the foggy scene image corresponding to the source domain image, which can improve the generation quality of the foggy scene image.

[0090] Optionally, the depth estimation model may be configured according to actual conditions, and this embodiment does not impose any specific limitation on this.

[0091] For example, the above-mentioned depth estimation model can be depthanything, that is, the source domain image is input into the ViT-based DINOv2 encoder of depthanything to extract high-dimensional features, and the high-dimensional features are passed to the DPT-based decoder of depthanything, which maps the features into a depth prediction map, that is, depth information.

[0092] Optionally, the fog scattering model follows the following formula (1):

[0093] I(x)=J(x)t(x)+A(1-t(x)) (1);

[0094] In formula (1), I(x) represents the pixel value of the foggy scene image corresponding to the source domain image; J(x) represents the pixel value of the source domain image; A represents the atmospheric light value; t(x) represents the transmittance of the source domain image, which is defined as the light transmission ratio from the scene point to the camera. The relationship between the transmittance and the depth information of the source domain image follows the following formula (2):

[0095] t(x)=e -βd (x) (2);

[0096] In formula (2), d(x) represents the depth information of the source domain image; β represents the preset medium scattering coefficient. The larger the value of the medium scattering coefficient, the higher the fog density of the foggy scene image corresponding to the source domain image. It can be understood that, according to actual needs, the medium scattering coefficients corresponding to different source domain images can be the same or different, and can be set according to actual conditions.

[0097] In terms of night scene image generation, a generative adversarial network is used to convert source domain images into night scene images corresponding to the source domain images, which can improve the generation quality of night scene images.

[0098] Optionally, the above-mentioned generative adversarial network can be set according to actual conditions, and this embodiment does not specifically limit this.

[0099] For example, the above-mentioned generative adversarial network can be CycleGAN, which uses unsupervised learning and combines paired adversarial loss and cycle consistency loss to achieve high-quality image translation between different domains. Specifically, CycleGAN may include a generator G, a generator F, a discriminator DX, and a discriminator DY. Generator G is used to convert source domain images into target domain images, generator F is used to convert target domain images into source domain images, and discriminator DX and discriminator DY are both used to determine whether the input image is real. Discriminator DX targets the source domain, and discriminator DY targets the target domain. The goal of CycleGAN is to achieve training through adversarial loss and cycle consistency loss. The CycleGAN trained in this way has the ability to transfer style in the image domain. By inputting the source domain image into CycleGAN, the night scene image corresponding to the source domain image can be generated.

[0100] In some embodiments, reference Figure 2 In the above step S300, the implementation process of performing model training and label generation processing according to multiple source domain images and multiple target domain images to obtain the first detection model and the target domain image set may include:

[0101] Based on the preset teacher network, multiple target domain images are inferred and processed to obtain several initial candidate boxes of each target domain image;

[0102] According to several initial candidate boxes of each target domain image, a pseudo label of each target domain image is obtained;

[0103] The preset student network is trained according to the training set of the i-th batch; wherein the training set of the i-th batch includes a plurality of target domain images with pseudo labels and source domain images belonging to the i-th batch and with manual labels;

[0104] Update the model parameters of the student network using the supervised loss and unsupervised loss of the student network;

[0105] When i<N, N is a first preset threshold value. If the i-th batch does not meet the preset update condition, then i=i+1, and return to the step of training the preset student network according to the training set of the i-th batch, otherwise update the model parameters of the teacher network, and return to the step of performing reasoning processing on multiple target domain images based on the preset teacher network until i≥N, obtain the target domain image set according to each target domain image and its pseudo label, and determine the student network after the model parameters are updated as the first detection model;

[0106] The above steps start from i=1.

[0107] In this embodiment, a domain adaptation model based on a teacher network-student network architecture is proposed for training a cross-domain object detection model, wherein both the teacher network and the student network include an encoder and a prediction head. For ease of understanding, this training process is also referred to as the first stage training.

[0108] Specifically, first, multiple target domain images are input into a preset teacher network, and the teacher network is set to inference mode. The teacher network performs an inference process on multiple target domain images, and the results of the detection inference are saved as several initial candidate boxes of the multiple target domain images. It is worth noting that among the multiple target domain images input into the teacher network, only the first frame target domain image has an artificial label, and the other target domain images do not have any labels. In addition, in order to improve the performance of the teacher network, before training, the model parameters obtained by training based on multiple source domain images can be used to initialize the teacher network.

[0109] Optionally, the number of initial candidate boxes can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the number of initial candidate boxes is 25, but it is not limited thereto.

[0110] Then, for each target domain image, several initial candidate frames of the target domain image are screened, aiming to select the candidate frame with the highest quality from all the initial candidate frames as the pseudo label of the target domain image. By traversing all target domain images, the pseudo labels of all target domain images can be obtained.

[0111] Afterwards, multiple source domain images with artificial labels are divided into N batches to obtain source domain images belonging to the N batches, and the source domain images belonging to the N batches are integrated with all target domain images with pseudo labels to obtain N batches of training sets. Among them, the training set of the i-th batch can include all target domain images with pseudo labels and source domain images belonging to the i-th batch and with artificial labels, i = 1, 2, 3, ..., N. It can be understood that there is at least one source domain image belonging to the i-th batch. After completing the division of the training set, for the training of the i-th batch, the training set of the i-th batch is input into the student network, that is, the student network simultaneously receives all target domain images with pseudo labels and source domain images belonging to the i-th batch and with artificial labels as input, and the two jointly supervise the training process of the student network.

[0112] It is worth noting that in this training process, the student network uses source domain images with artificial labels for supervised learning, and uses target domain images with pseudo labels for unsupervised learning.

[0113] After the training of the i-th batch is completed, the supervised loss and unsupervised loss of the student network are calculated, and the supervised loss and unsupervised loss of the student network are used to update the model parameters of the student network to obtain the updated model parameters of the student network. It can be understood that the model parameters can be set according to actual conditions. For example, the model parameters may include network hyperparameters such as bias and weight of neurons, but are not limited thereto.

[0114] Finally, determine whether i<N holds.

[0115] If i<N holds, it means that the first stage of training has not yet ended. At this time, it is further determined whether the i-th batch meets the preset update conditions to determine whether the teacher network needs to be updated. If it is determined that the i-th batch meets the update conditions, it means that the teacher network needs to be updated. At this time, the model parameters of the teacher network are updated, and the updated model parameters of the teacher network are obtained as the new model parameters of the teacher network, and the step of performing reasoning processing on multiple target domain images based on the preset teacher network is returned to update the pseudo-labels generated by the teacher network, thereby improving the quality of the pseudo-labels. If it is determined that the i-th batch does not meet the update conditions, it means that the teacher network does not need to be updated for the time being. At this time, the updated model parameters of the student network are used as the new model parameters of the student network, and i=i+1 is set, and the step of training the preset student network according to the training set of the i-th batch is returned to realize iterative training.

[0116] If i<N does not hold, it means that the first stage of training can be ended. At this time, each target domain image is used as a sample, and the pseudo-label of each target domain image is used as a label. Each target domain image and its pseudo-label are integrated into a target domain image set. At the same time, the student network after the model parameters are updated is determined as the first detection model, thereby completing the first stage of training.

[0117] It can be seen that this embodiment provides a domain adaptation model based on the teacher-student architecture, which is used for the training of cross-domain target detection models. It can not only improve the integrity of the target domain image and reduce the label annotation cost, but also transfer the knowledge of non-extreme scenes to extreme scenes, so that the detection model can fully learn the knowledge of different domains, thereby ensuring that the detection model is adequately trained, reducing the possibility of problems such as overfitting, and improving the robustness of the detection model in extreme scenes.

[0118] Optionally, the encoder and prediction head configured for the teacher network and the student network can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the teacher network and the student network both use the same Vision Transformer as an encoder and the same Center Head as a prediction head, but are not limited to this.

[0119] Optionally, the first preset threshold may be set according to actual conditions, and this embodiment does not specifically limit this. For example, the first preset threshold may be 250, that is, the first stage training includes 250 batches of training, but is not limited thereto.

[0120] Optionally, the above update conditions can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above update condition can be that the i-th batch is a batch used to update the teacher network. Specifically, during the first stage of training, this embodiment updates the teacher network once every v rounds, and the value of v can be set according to actual conditions. For example, v can be 5. Based on this, multiple batches for updating the teacher network can be selected from N batches to obtain a batch set. If the i-th batch is a batch used to update the teacher network, that is, the i-th batch belongs to the batch set, then it is determined that the i-th batch meets the update condition, otherwise it is determined that the i-th batch does not meet the update condition.

[0121] In some implementations, obtaining a pseudo label of each target domain image according to a plurality of initial candidate boxes of each target domain image may include:

[0122] Screening several initial candidate frames of the target domain image to obtain multiple target candidate frames of the target domain image;

[0123] Voting is performed on each target candidate box of the target domain image to obtain a voting weight of each target candidate box of the target domain image;

[0124] According to the coordinate position and voting weight of each target candidate box in the target domain image, the pseudo label of the target domain image is obtained.

[0125] In this embodiment, the appearance of the target in extreme scenes often changes dramatically. In this regard, pseudo labels are usually used to improve the adaptability of the target domain. However, since the generated pseudo labels are not screened with high quality, the quality of the pseudo labels is low and contains a lot of noise. Although there are a few methods to screen the generated pseudo labels, they do not optimize and screen the output of the model, and the quality of the pseudo labels still needs to be improved.

[0126] In this regard, this embodiment proposes a pseudo-label voting mechanism to improve the quality of pseudo-labels. The pseudo-label mechanism first selects the top K candidate positions with the highest response values ​​through the classification score heat map, and calculates the confidence of each position based on the Gaussian weight soft voting mechanism to measure the reliability of the pseudo-label. Then, positions with higher confidence will be given higher weights, while positions with lower confidence will be given lower weights. Finally, the final position of the pseudo-label is adjusted through weighted voting. In this way, low-quality pseudo-labels can be effectively filtered and the positioning error caused by pseudo-label noise can be reduced, thereby effectively improving the quality of pseudo-labels.

[0127] In the pseudo-label voting mechanism, first, the prediction head of the teacher network performs a convolution operation on the feature information extracted by the encoder of the teacher network to obtain the component features of each initial candidate box of the target domain image. The component features include classification score, scale size and offset. Based on the component features of each initial candidate box, several initial candidate boxes of the target domain image are screened to obtain multiple target candidate boxes of the target domain image. Specifically, based on the classification score, several initial candidate boxes are sorted from large to small and the first K initial candidate boxes are selected as target candidate boxes, as shown in the following formula (3):

[0128]

[0129] In formula (3), S topK represents the set consisting of the first K initial candidate boxes; TopK(·) represents the operation of selecting the first K initial candidate boxes; φ(·) represents the embedding function of the teacher network; Represents a template; represents the search area; θ T Represents the model parameters of the teacher network. It should be noted that in the single target tracking network model, the input image is cropped in two ways according to the initial annotation box, where the smaller one is the template, which represents the region of interest, and the larger one is the search area, which represents the region of interest and the Beijing area. The model can accept these two inputs at the same time and find targets similar to the template in the search area and select them as tracking results, namely candidate boxes.

[0130] Optionally, the number of target candidate frames can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the number of target candidate frames can be 10, but is not limited thereto.

[0131] Then, a pseudo-label voting strategy is performed for the selected K target candidate boxes, which follows the following formula (4):

[0132]

[0133] In formula (4), M p represents the voting weight of the p-th target candidate box; C p represents the classification score of the pth target candidate box; C min represents the minimum classification score of K target candidate boxes; C max represents the maximum classification score of K target candidate boxes; D(C p , C max ) represents the distance between the classification score of the pth target candidate box and the maximum classification score of the K target candidate boxes; σ p It refers to the standard deviation of the p-th target candidate box obtained by the K-Means clustering algorithm.

[0134] It is worth noting that in the pseudo-label voting strategy, the K-Means clustering algorithm is used to divide the classification scores into different clusters based on similarity, which refers to the distance between different classification scores or the similarity of features. Specifically, the K-Means clustering algorithm measures the similarity by calculating the Euclidean distance between each classification score and the cluster center, and then calculates the standard deviation of the points within each cluster. The clustering algorithm can be used to assign classification scores to similar clusters. The standard deviation of each cluster reflects the degree of discreteness of the points within the cluster, that is, the distance between the points and the cluster center. This standard deviation is used to measure the reliability of each cluster, thereby helping to evaluate the accuracy and stability of pseudo-labels.

[0135] Finally, based on the coordinate position and voting weight of each target candidate box in the target domain image, a weighted calculation is performed to obtain the pseudo label of the target domain image, which is shown in the following formula (5):

[0136]

[0137] In formula (5), B * represents the pseudo label of the target domain image; B p represents the coordinate position of the pth target candidate box, which can be expressed as B p ={x1, y1, x2, y2}.

[0138] In some implementations, the above-mentioned updating of the model parameters of the student network using the supervised loss and the unsupervised loss of the student network may include:

[0139] According to the predicted labels and manual labels of the source domain images belonging to the i-th batch, the supervision loss of the student network is obtained; wherein the predicted labels of the source domain images are the label information obtained by predicting the source domain images by the student network;

[0140] According to the predicted labels and pseudo labels of each target domain image, the unsupervised loss of the student network is obtained; wherein the predicted label of the target domain image is the label information obtained by predicting the target domain image by the student network;

[0141] According to the supervised loss and unsupervised loss of the student network, the target loss of the student network is obtained;

[0142] Backpropagation is performed using the target loss of the student network to update the model parameters of the student network.

[0143] In this embodiment, the student network is trained using source domain data and target domain data, wherein the student network uses source domain images and their artificial labels for supervised learning, and uses target domain images and their pseudo labels for unsupervised learning. Therefore, in the first stage of training, two loss functions, supervised learning loss function and unsupervised learning loss function, are used as the loss function of the student network, so that the effect of consistency loss can be achieved. Through the common effect of the above-mentioned supervised loss and unsupervised loss, the model can correspond the tracking knowledge of the source domain with the tracking knowledge in extreme scenarios, so that the model can cope with various extreme scenarios and has higher robustness, that is, the model has "consistent" tracking performance for various scenarios.

[0144] For supervised learning loss, the predicted labels of the source domain images belonging to the i-th batch can be obtained through the prediction of the student network. Then, the predicted labels of the source domain images belonging to the i-th batch and the manual labels are used to calculate the classification loss function, regression loss function and intersection-over-union loss function to obtain the supervised loss of the student network, which is shown in the following formula (6):

[0145]

[0146] In formula (6), L track represents the supervision loss of the student network; L cls represents the supervised classification loss of the student network; L iou Represents the supervised IoU loss of the model positioning box of the student network, that is, the supervised intersection-over-union loss; λ iou represents the weight of the supervised intersection-over-union loss; L1 represents the supervised regression loss of the student network, which uses the L1 loss function, i.e., the mean absolute error (MAE) loss function; represents the weight of the supervised regression loss.

[0147] Optionally, the above classification loss function can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above classification loss function can be a cross entropy loss function, but is not limited thereto.

[0148] Optionally, the weight of the supervised regression loss and the weight of the supervised intersection-over-union loss are set according to actual conditions, and this embodiment does not specifically limit this. For example, the weight of the supervised regression loss may be 5, and the weight of the supervised intersection-over-union loss may be 2, but is not limited thereto.

[0149] The unsupervised learning loss is roughly the same as the supervised learning loss. The difference is that the unsupervised learning loss uses pseudo labels and their corresponding images. Specifically, the predicted labels of each target domain image can be obtained through the prediction of the student network, and the predicted labels and pseudo labels of each target domain image are used to calculate the classification loss function, regression loss function and intersection-over-union loss function to obtain the unsupervised loss of the student network. Its expression is the same as the above formula (6), which will not be repeated here.

[0150] Afterwards, the supervised loss and unsupervised loss of the student network are weighted to obtain the target loss of the student network, which is shown in the following formula (7):

[0151] L=L track +λL untrack (7);

[0152] In formula (7), L represents the target loss of the student network; L track represents the supervision loss of the student network; L untrack represents the unsupervised loss of the student network; λ represents the weight of the unsupervised loss.

[0153] Optionally, the weight of the unsupervised loss can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the weight of the unsupervised loss can be 0.5, but is not limited thereto.

[0154] Finally, the target loss of the student network is used for back propagation to update the model parameters of the student network, which can ensure the training accuracy of the student network and thus improve the detection accuracy of the first detection model.

[0155] In some implementations, updating the model parameters of the teacher network may include:

[0156] According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

[0157] In this embodiment, since the teacher network is set to inference mode, the teacher network does not actively update its model parameters. In this regard, this embodiment uses the exponential moving average method (EMA) and combines the updated model parameters of the student network to update the model parameters of the teacher network to iteratively improve the quality of the pseudo labels it generates. The above process is shown in the following formula (8):

[0158] v t =αv t-1 +(1-α)θ t (8);

[0159] In formula (8), v t represents the updated model parameters of the teacher network in the tth update; v t-1 represents the average value of all updated model parameters of the teacher network in the first t-1 times; θ t represents the updated model parameters of the student network before the tth update; α is the equilibrium constant, which is used to adjust the weight parameter of the average of the previous parameters. It should be noted that the tth update in this formula refers to the tth update of the teacher network.

[0160] Optionally, the above equilibrium constant can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above equilibrium constant can be 0.9, but is not limited thereto.

[0161] In some embodiments, reference Figure 2 In the above step S400, training the first detection model according to the target domain image set to obtain the second detection model may include:

[0162] According to the first detection model, a third detection model is obtained;

[0163] The third detection model is trained according to the training set of the jth batch; wherein the training set of the jth batch includes target domain images belonging to the jth batch and having pseudo labels;

[0164] Updating model parameters of the third detection model using the unsupervised loss of the third detection model;

[0165] When j<N', set j=j+1, and return to the step of training the third detection model according to the j-th batch of training sets until j≥N', and determine the third detection model after the model parameters are updated as the second detection model;

[0166] The above steps start from j=1.

[0167] In this embodiment, a third detection model can be obtained based on the first detection model. Exemplarily, the first detection model is directly determined as the third detection model. Also exemplarily, some modules for improving the performance of the model are added to the first detection model to obtain a third detection model. These modules can be existing modules, such as modules based on a certain attention mechanism, or non-existing modules, but are not limited to this. It is worth noting that in the embodiment of the present application, a dynamic aggregation prompt module is added to the first detection model to improve the performance of the detection model. Its specific implementation will be described in the following embodiments. After obtaining the third detection model, the target domain image set obtained in the above steps is used to train the third detection model. This training process can be called the second stage training.

[0168] In the second stage of training, first, the target domain image set is divided into N' batches to obtain N' batches of training sets. Among them, the training set of the jth batch may include target domain images belonging to the jth batch and having pseudo labels, j = 1, 2, 3, ..., N'. It can be understood that there is at least one target domain image belonging to the jth batch and having a pseudo label. Then, for the training of the jth batch, the training set of the jth batch is input into the third detection model, that is, the third detection model receives the target domain image belonging to the jth batch and having a pseudo label and is trained based on this to learn the relationship between the target domain image and its pseudo label. After the training of the jth batch is completed, the unsupervised loss of the third detection model is calculated, and the unsupervised loss of the third detection model is used for back propagation to update the model parameters of the third detection model to obtain the updated model parameters of the third detection model. It can be understood that the specific implementation of calculating the unsupervised loss of the third detection model is the same as the specific implementation of calculating the unsupervised loss of the student network in the aforementioned embodiment, which will not be repeated. Finally, it is determined whether j<N' is established. If j<N' holds, it means that the second stage training has not yet been completed. At this time, the updated model parameters of the third detection model are determined as the new model parameters of the third detection model, and j=j+1 is set, and the step of training the third detection model according to the jth batch of training sets is returned to realize iterative training. If j<N' does not hold, it means that the second stage training can be completed. At this time, the third detection model with updated model parameters is determined as the second detection model.

[0169] It can be seen that in this embodiment, the first detection model is trained using target domain images with pseudo labels, so that the detection model can fully learn the mapping relationship between the target domain images and their labels, thereby having the ability to perform target detection tasks in extreme scenes, which can effectively improve the performance of the detection model in target detection tasks in extreme scenes.

[0170] Optionally, the second preset threshold can be flexibly set according to actual conditions, and this embodiment does not specifically limit this. For example, the second preset threshold is 50, that is, the second stage training includes 50 batches of training, but it is not limited thereto.

[0171] In some embodiments, reference Figure 3 , the above-mentioned second detection model may include an input layer, a feature extraction structure and a prediction head, wherein:

[0172] The input layer is used to process the image to be tested into blocks to obtain an image block sequence;

[0173] The feature extraction structure includes a plurality of feature extraction modules connected in sequence, the input feature of the first feature extraction module is an image block sequence, the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to extract features from the input features of the feature extraction module to obtain the search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain the output features of the feature extraction module according to the search features and input features of the feature extraction module;

[0174] The prediction head is used to make predictions based on the output features of the last feature extraction module to obtain the target detection results of the image to be tested.

[0175] In this embodiment, it can be seen from the above embodiments that the first detection model is a trained student network, and in some examples, the student network can use Vision-Transformer as an encoder and Center Head as a prediction head. Based on this, the first detection model of this embodiment can use Vision-Transformer as the basic network and Center Head as the prediction head. In order to improve the performance of the detection model, a dynamic aggregation prompt module is added between each layer of Transformer encoder of Vision-Transformer to obtain a third detection model. The second detection model can be obtained by training the third detection model.

[0176] In the real-time target detection task, the image to be tested is input into the second detection model. In the second detection model, first, the image to be tested is divided into blocks through the input layer and position encoding is added to obtain an image block sequence, which is input into the feature extraction structure; then, multiple feature extraction modules of the feature extraction structure are used to perform multiple rounds of feature extraction on the image block sequence; finally, the output features of the last feature extraction module are predicted through the Center Head prediction head to obtain the target detection result of the image to be tested, thereby realizing the target detection task in extreme scenarios.

[0177] Specifically, the feature extraction structure may include a plurality of feature extraction modules connected in sequence, the input features of the first feature extraction module are image block sequences, and the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module. Each feature extraction module includes a Transformer encoder and a dynamic aggregation prompt module. In each feature extraction module, the input features of the feature extraction module are subjected to feature extraction by the Transformer encoder to obtain the search features of the feature extraction module, and then the search features and input features of the feature extraction module are aggregated by the dynamic aggregation prompt module to obtain the output features of the feature extraction module. This can improve the feature extraction capability of the second detection model in extreme scenarios, thereby improving the target detection accuracy of the second detection model in extreme scenarios.

[0178] Optionally, the number of the feature extraction modules can be set according to actual conditions, and this embodiment does not specifically limit this.

[0179] In some implementations, in the above-mentioned dynamic aggregation prompt module, the implementation process of obtaining the output features of the feature extraction module according to the search features and input features of the feature extraction module may include:

[0180] According to the search features and input features of the feature extraction module, a fusion feature of the feature extraction module is obtained;

[0181] According to the search features and input features of the feature extraction module, a dynamic prompt of the feature extraction module is obtained;

[0182] According to the dynamic prompt and search features of the feature extraction module, the output features of the feature extraction module are obtained.

[0183] In this embodiment, the dynamic aggregation prompt module aims to enhance the detection model's adaptability to changes in target appearance through a dynamic prompt mechanism. It first fuses the search features of the current stage with the updated search features of the previous stage and balances their weights to generate hierarchical visual features, which can better capture the contextual feature information of the target. Then, the global context feature information of the search features of the current stage and the updated search features of the previous stage is extracted, and the global context feature information is encoded to generate dynamic prompts with scene adaptability. Finally, the dynamic prompt is used to update the search features of the current stage to obtain the final output features, which can further enhance the model's modeling ability for changes in target appearance and its robustness to interference from extreme scenes.

[0184] Specifically, in order to fuse the hierarchical visual semantic information and obtain hierarchical visual features, this embodiment first uses the neighbor search features for fusion. Specifically, according to the search features and input features of the feature extraction module, the fusion features of the feature extraction module are obtained, as shown in the following formula (9):

[0185] X′ l =G(X l ;θ)⊙X l +(1-G(X l-1 ;θ))⊙X l-1 (9);

[0186] In formula (9), X′ l represents the fusion feature of the lth feature extraction module; X l represents the search feature of the lth feature extraction module; X l-1 represents the input features of the lth feature extraction module, that is, the output features of the l-1th feature extraction module, which can be understood as the updated search features of the l-1th feature extraction module; G(·) represents the Sigmoid gating function with bilinear upsampling, which is used to adjust the feature fusion weight; θ represents the preset learnable parameter, which can be flexibly set according to the actual situation; ⊙ represents element-by-element multiplication. This process extracts hierarchical visual semantic information by layering the current search features and the updated search features of the previous layer, which can help the model adapt to the appearance changes of the target object.

[0187] At the same time, in order to enhance the adaptability of the model to changes in target appearance, this embodiment proposes a dynamic prompt for the aggregation of search features. Specifically, according to the search features and input features of the feature extraction module, the dynamic prompt of the feature extraction module is obtained, as shown in the following formula (10):

[0188]

[0189] In formula (10), P l Represents the dynamic prompt of the lth feature extraction module; Represents X l The u-th element in ; Represents X l-1 The u-th element in ; Represents preset learnable parameters, which can be flexibly set according to actual conditions; Represents the sequentially connected convolutional layers and pooling layers. The pooling operation type of the pooling layer can be flexibly set according to the actual situation. For example, the pooling operation can be a global average pooling operation, but it is not limited to this. This process assigns corresponding weights to each search feature through the attention mechanism. Dynamic prompts can enable the model to effectively capture the changing pattern of the target appearance.

[0190] Finally, the generated dynamic prompts are used to adjust the target features extracted by the model by multiplying the corresponding elements. Specifically, the search features of the feature extraction module are updated using the dynamic prompts of the feature extraction module, so as to obtain the updated search features of the feature extraction module as the output features of the feature extraction module, as shown in the following formula (11):

[0191] X l =X l ·P l (11).

[0192] To facilitate the understanding of the domain adaptive target detection method in the above extreme scenarios in the embodiment of the present application, the domain adaptive target detection method in the above extreme scenarios in the embodiment of the present application will be described below using an application scenario. In this application scenario, the target domain is a foggy scene and a night scene (i.e., an extreme scene), and the source domain is a daytime scene (i.e., a non-extreme scene). The domain adaptive target detection method in the above extreme scenes can be divided into four parts: image synthesis, first-stage training, second-stage training, and real-time target detection, which is specifically implemented as follows:

[0193] S01, Image Synthesis:

[0194] First, multiple daytime scene images are obtained as source domain images. For each daytime scene image, the daytime scene image is first input into the ViT-based DINOv2 encoder of depthanything to extract high-dimensional features. The high-dimensional features are passed to the DPT-based decoder of depthanything, which maps the features into a depth prediction map, i.e., depth information. Then, the daytime scene image and its depth information are combined with the fog scattering model shown in the above formulas (1)-(2) to obtain the foggy scene image corresponding to the daytime scene image; at the same time, the daytime scene image is input into the trained CycleGAN to generate the nighttime scene image corresponding to the source domain image. Then, only the artificial label of the first daytime scene image is assigned to the foggy scene image and the nighttime scene image corresponding to the first daytime scene image, while the foggy scene images and the nighttime scene images corresponding to the other daytime scene images do not have any labels. In this way, multiple foggy scene images and multiple nighttime scene images can be obtained as target domain images.

[0195] S02, the first stage of training, which includes 250 batches of training:

[0196] First, a teacher network and a student network are constructed. Both the teacher network and the student network use the same VisionTransformer as the encoder and the same Center Head as the prediction head. Before training, the teacher network is initialized using the model parameters obtained by training based on multiple source domain images.

[0197] Then, before the i-th batch training, the teacher network is used to perform the reasoning process on multiple target domain images, and the detection reasoning results are saved as 25 initial candidate boxes of multiple target domain images. For each target domain image, the 25 initial candidate boxes are first sorted from large to small based on their classification scores, and the first 10 initial candidate boxes are selected as target candidate boxes, which is implemented as shown in the above formula (3). Then, a pseudo-label voting strategy is performed for the selected 10 target candidate boxes to obtain the voting weights of the 10 target candidate boxes, which follows the above formula (4). Finally, a weighted calculation is performed based on the coordinate positions and voting weights of the 10 target candidate boxes to obtain the pseudo-labels of the target domain images, which is implemented as shown in the above formula (5). By traversing all target domain images, the pseudo-labels of all target domain images can be obtained.

[0198] Afterwards, in the i-th batch training, the training set of the i-th batch is input into the student network, that is, the student network simultaneously receives all the target domain images with pseudo labels and the source domain images belonging to the i-th batch and with manual labels as input, and the two jointly supervise the training process of the student network. After the i-th batch training is completed, the target loss of the student network is calculated. The target loss is composed of the unsupervised loss and the supervised loss, as shown in the above formulas (6)-(7), and the target loss of the student network is used for back propagation to update the model parameters of the student network.

[0199] Finally, determine whether i<N holds, where N is 250.

[0200] If i<N holds, it is further determined whether the i-th batch is the batch used to update the teacher network.

[0201] If the i-th batch is determined to be a batch for updating the teacher network, the model parameters of the teacher network are updated based on the updated model parameters of the student network combined with the exponential moving average method, which is implemented as shown in the above formula (8), and the updated model parameters of the teacher network are obtained as the new model parameters of the teacher network, and the process returns to the step of performing the reasoning process on multiple target domain images through the teacher network to update the pseudo labels generated by the teacher network, thereby improving the quality of the pseudo labels. In this application scenario, the teacher network is updated once every 5 rounds, that is, before the 1st round of training, the 6th round of training, the 11th round of training, ..., and the 246th round of training.

[0202] If it is determined that the i-th batch is not the batch used to update the teacher network, the updated model parameters of the student network are used as the new model parameters of the student network, i=i+1 is set, and the training step of the i-th batch is returned to realize iterative training.

[0203] If i<N does not hold, each target domain image is used as a sample, and the pseudo-label of each target domain image is used as a label. Each target domain image and its pseudo-label are integrated into a target domain image set. At the same time, the student network with updated model parameters is determined as the first detection model, thereby completing the first stage of training.

[0204] S03, the second stage of training, which includes 50 batches:

[0205] First, the first detection model uses Vision-Transformer as the basic network and Center Head as the prediction head. In order to improve the performance of the detection model, a dynamic aggregation prompt module is added between each layer of Transformer encoder of Vision-Transformer to obtain the third detection model.

[0206] Then, the target domain image set is divided into N' batches to obtain N' batches of training sets, wherein the j-th batch of training sets may include target domain images belonging to the j-th batch and having pseudo labels.

[0207] Afterwards, in the training of the jth batch, the training set of the jth batch is input into the third detection model, that is, the third detection model receives the target domain images belonging to the jth batch and having pseudo labels, and is trained based on these target domain images to learn the relationship between the target domain images and their pseudo labels. After the training of the jth batch is completed, the unsupervised loss of the third detection model is calculated, and the unsupervised loss of the third detection model is used for back propagation to update the model parameters of the third detection model, and obtain the updated model parameters of the third detection model.

[0208] Finally, it is determined whether j<N' is established, and N' is 50. If j<N' is established, the updated model parameters of the third detection model are determined as the new model parameters of the third detection model, and j=j+1 is set, and the step of training the third detection model according to the training set of the jth batch is returned to realize iterative training. If j<N' is not established, the third detection model after the model parameters are updated is determined as the second detection model.

[0209] S04, Real-time Object Detection:

[0210] First, the input layer divides the image to be tested into blocks and adds position encoding to obtain an image block sequence, which is input into the feature extraction structure. Then, the image block sequence is subjected to multiple rounds of feature extraction through multiple feature extraction modules of the feature extraction structure. Finally, the output features of the last feature extraction module are predicted through the Center Head prediction head to obtain the target detection result of the image to be tested, such as vehicles.

[0211] Specifically, the feature extraction structure includes a plurality of feature extraction modules connected in sequence, the input features of the first feature extraction module are the image block sequence, and the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module. Each feature extraction module includes a Transformer encoder and a dynamic aggregation prompt module. In each feature extraction module, the input features of the feature extraction module are first extracted by the Transformer encoder to obtain the search features of the feature extraction module, and then the search features of the feature extraction module and the input features are aggregated using the dynamic aggregation prompt module to obtain the output features of the feature extraction module.

[0212] More specifically, in the dynamic aggregation prompt module, first, according to the search features and input features of the feature extraction module, the fusion features of the feature extraction module are obtained, as shown in the above formula (9); then, according to the search features and input features of the feature extraction module, the dynamic prompts of the feature extraction module are obtained, as shown in the above formula (10); finally, according to the dynamic prompts and search features of the feature extraction module, the output features of the feature extraction module are obtained, as shown in the above formula (11).

[0213] In order to further verify the effect of the embodiment of the present application, the embodiment of the present application was verified on the dataset Foggy-Cityscapes and the dataset BDD100K. After verification, the embodiment of the present application demonstrated excellent performance in extreme scenes of foggy days and nighttime, reaching the current SOTA level. Specifically, the model effectively alleviates the problem of semantic offset between the source domain and the target domain by optimizing the quality of pseudo-labels and feature alignment capabilities, and exhibits excellent accuracy and robustness in a variety of complex scenarios. At the same time, the dynamic aggregation prompt module enhances the model's ability to adapt to drastic changes in target appearance, so that it can still maintain stable target detection performance in an interfering background. The mAP of the embodiment of the present application on the dataset Foggy-Cityscapes reaches 60.5%, and the mAP detection performance of targets in low-quality images on the dataset BDD100K reaches 63.1%. In addition, the model has extremely high real-time detection performance, running at a speed of 110FPS on GPU devices and 38FPS on embedded devices. It can not only accurately complete target positioning tasks, but also meet the real-time requirements of target detection in actual complex environments, fully verifying the practicality and reliability of the model.

[0214] In addition, refer to Figure 3 The present application also provides a domain adaptive target detection device in an extreme scenario, which may include:

[0215] An acquisition module 101 is used to acquire a plurality of source domain images with artificial labels, where the source domain images are non-extreme scene images;

[0216] A first processing module 102 is used to obtain multiple target domain images according to multiple source domain images, where the target domain images are extreme scene images, and only the first target domain image has an artificial label, while the other target domain images have no labels;

[0217] A second processing module 103 is used to perform model training and label generation processing according to multiple source domain images and multiple target domain images to obtain a first detection model and a target domain image set, where the target domain image set includes multiple target domain images with pseudo labels;

[0218] The third processing module 104 is used to train the first detection model according to the target domain image set to obtain a second detection model;

[0219] The fourth processing module 105 is used to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

[0220] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0221] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0222] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A domain adaptive object detection method in extreme scenarios, characterized in that: The following steps are involved: Acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images; According to the plurality of source domain images, a plurality of target domain images are obtained; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels; Performing model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels; Training the first detection model according to the target domain image set to obtain a second detection model; Perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

2. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The step of obtaining a plurality of target domain images according to the plurality of source domain images comprises: The image generation process of the extreme scene is performed based on each of the source domain images to obtain the target domain image corresponding to each of the source domain images, and the artificial label of the first source domain image is assigned to the target domain image corresponding to the first source domain image, thereby obtaining a plurality of the target domain images.

3. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The performing model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set includes: Performing reasoning processing on the plurality of target domain images based on a preset teacher network to obtain a plurality of initial candidate frames of each of the target domain images; Obtaining the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image; The preset student network is trained according to the i-th batch of training sets; wherein the i-th batch of training sets includes a plurality of the target domain images with the pseudo labels and the source domain images belonging to the i-th batch and having the artificial labels; Updating the model parameters of the student network using the supervised loss and the unsupervised loss of the student network; When i<N, N is a first preset threshold value, if the i-th batch does not meet the preset update condition, then set i=i+1, and return to the step of training the preset student network according to the training set of the i-th batch, otherwise update the model parameters of the teacher network, and return to the step of performing inference processing on the plurality of target domain images based on the preset teacher network, until i≥N, obtain the target domain image set according to each of the target domain images and their pseudo labels, and determine the student network after the model parameters are updated as the first detection model; The above steps start from i=1.

4. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The step of obtaining the pseudo label of each target domain image according to the plurality of initial candidate frames of each target domain image comprises: Screening the plurality of initial candidate frames of the target domain image to obtain a plurality of target candidate frames of the target domain image; Performing voting processing on each of the target candidate boxes in the target domain image to obtain a voting weight of each of the target candidate boxes in the target domain image; The pseudo label of the target domain image is obtained according to the coordinate position and voting weight of each target candidate box of the target domain image.

5. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The method of updating the model parameters of the student network using the supervised loss and the unsupervised loss of the student network comprises: Obtaining the supervision loss of the student network according to the predicted labels of the source domain images belonging to the i-th batch and the manual labels; wherein the predicted labels of the source domain images are label information obtained by predicting the source domain images by the student network; According to the predicted labels of each of the target domain images and the pseudo labels, an unsupervised loss of the student network is obtained; wherein the predicted labels of the target domain images are label information obtained by predicting the target domain images by the student network; Obtaining a target loss of the student network according to the supervised loss and the unsupervised loss of the student network; Back-propagation is performed using the target loss of the student network to update the model parameters of the student network.

6. The domain adaptive target detection method in extreme scenarios according to claim 3, characterized in that: The updating of the model parameters of the teacher network comprises: According to the updated model parameters of the student network, the model parameters of the teacher network are updated in combination with the exponential moving average method.

7. The domain adaptive object detection method in extreme scenarios according to claim 1, characterized in that: The step of training the first detection model according to the target domain image set to obtain a second detection model includes: According to the first detection model, a third detection model is obtained; The third detection model is trained according to the j-th batch of training sets; wherein the j-th batch of training sets includes the target domain images belonging to the j-th batch and having the pseudo labels; Updating model parameters of the third detection model using the unsupervised loss of the third detection model; When j<N', N' is a second preset threshold, let j=j+1, and return to the step of training the third detection model according to the jth batch of training sets until j≥N', and determine the third detection model after the model parameters are updated as the second detection model; The above steps start from j=1.

8. The domain adaptive target detection method in extreme scenarios according to claim 1, characterized in that: The second detection model includes an input layer, a feature extraction structure and a prediction head, wherein: The input layer is used to process the image to be tested in blocks to obtain an image block sequence; The feature extraction structure includes a plurality of feature extraction modules connected in sequence, the input feature of the first feature extraction module is the image block sequence, the input features of the feature extraction modules other than the first feature extraction module are the output features of the previous feature extraction module, the feature extraction module includes an encoder and a dynamic aggregation prompt module, the encoder is used to perform feature extraction on the input features of the feature extraction module to obtain the search features of the feature extraction module, and the dynamic aggregation prompt module is used to obtain the output features of the feature extraction module according to the search features and the input features of the feature extraction module; The prediction head is used to make predictions based on the output features of the last feature extraction module to obtain the target detection result of the image to be tested.

9. The domain adaptive object detection method in extreme scenarios according to claim 8, characterized in that: The step of obtaining the output feature of the feature extraction module based on the search feature and the input feature of the feature extraction module comprises: Obtaining a fusion feature of the feature extraction module according to the search feature and the input feature of the feature extraction module; Obtaining a dynamic prompt of the feature extraction module according to the search feature and the input feature of the feature extraction module; The output feature of the feature extraction module is obtained according to the dynamic prompt and the search feature of the feature extraction module.

10. A domain adaptive target detection device in extreme scenarios, characterized in that: include: An acquisition module, used to acquire a plurality of source domain images with artificial labels; wherein the source domain images are non-extreme scene images; A first processing module is used to obtain a plurality of target domain images according to the plurality of source domain images; wherein the target domain images are extreme scene images, only the first target domain image has an artificial label, and the remaining target domain images have no labels; A second processing module is used to perform model training and label generation processing according to the plurality of source domain images and the plurality of target domain images to obtain a first detection model and a target domain image set; wherein the target domain image set includes a plurality of target domain images with pseudo labels; A third processing module, configured to train the first detection model according to the target domain image set to obtain a second detection model; The fourth processing module is used to perform target detection on the image to be tested according to the second detection model to obtain a target detection result of the image to be tested.

Images