Target detection method and device based on domain self-adaption, equipment and medium

Abstract

The invention discloses a target detection method and device based on domain self-adaption, equipment and a medium. The target detection method based on domain adaptation is realized through a target detection model, the target detection model comprises a dynamic domain adaptation module, and the dynamic domain adaptation module comprises an image level domain classifier with a first dynamic confrontation gradient inversion layer and an object level domain classifier with a second dynamic confrontation gradient inversion layer. In the back propagation of the training process of the target detection model, the gradient inversion intensity of the first dynamic confrontation gradient inversion layer is dynamically adjusted according to the image level domain classification loss, and the gradient inversion intensity of the second dynamic confrontation gradient inversion layer in the back propagation is dynamically adjusted according to the object level domain classification loss; according to the method, the target detection precision under the severe weather condition can be improved, the target missing detection rate and the target false detection rate under the severe weather condition are reduced, and then the safety and the reliability of the automatic driving system are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a target detection method, apparatus, device, and medium based on domain adaptation. Background Technology

[0002] Currently, in autonomous driving systems, target detection tasks involving the identification of vehicles, pedestrians, and traffic signs mainly rely on deep learning-based supervised learning models. These models are typically trained and validated on datasets collected under normal weather conditions, such as sunny days. However, in real-world applications, autonomous vehicles inevitably encounter various adverse weather conditions such as rain, fog, snow, and sandstorms. Models trained on normal weather data exhibit significantly reduced detection performance under adverse weather conditions, leading to increased false positives, higher missed detection rates, and lower positioning accuracy, thus impacting the safety and reliability of the autonomous driving system. Summary of the Invention

[0003] The purpose of this invention is to provide a domain-adaptive target detection method, apparatus, device, and medium that solves the above-mentioned technical problems.

[0004] According to a first aspect of the present invention, a domain-adaptive target detection method is provided. The method is implemented through a target detection model, which includes a dynamic domain adaptation module. The dynamic domain adaptation module includes: an image-level domain classifier with a first dynamic adversarial gradient inversion layer and an object-level domain classifier with a second dynamic adversarial gradient inversion layer. The method includes:

[0005] A target detection model is trained using a sample set, which includes samples from the target domain and samples from the source domain. The target domain corresponds to a first weather condition, and the source domain corresponds to a second weather condition. The training process of the target detection model includes forward propagation and backward propagation based on the sample set. The forward propagation includes calculating the total loss of the target detection model, which includes image-level domain classification loss and object-level domain classification loss. In the backward propagation, the gradient reversal strength of the first dynamic adversarial gradient reversal layer is dynamically adjusted according to the image-level domain classification loss. In the backward propagation, the gradient reversal strength of the second dynamic adversarial gradient reversal layer is dynamically adjusted according to the object-level domain classification loss.

[0006] Under the first weather conditions, images captured by the camera mounted on the vehicle and point clouds captured by the radar mounted on the vehicle are acquired. The target detection model is then used to obtain the target detection result of the vehicle based on the images and point clouds.

[0007] In some embodiments of the first aspect of the present invention, the gradient reversal strength of the first dynamic adversarial gradient reversal layer and the gradient reversal strength of the first dynamic adversarial gradient reversal layer are both dynamically adjusted in the following manner: when the domain classification loss is less than a preset domain classification loss threshold, the gradient reversal strength is set to the smaller value between the quotient obtained by dividing the preset fixed value by the domain classification loss and the preset upper limit of the gradient reversal strength; and when the domain classification loss is greater than or equal to the domain classification loss threshold, the gradient reversal strength is set to the preset fixed value.

[0008] In some embodiments of the first aspect of the present invention, the training process of the target detection model includes a first stage, a second stage, and a third stage executed sequentially, wherein the domain classification loss of the sample set in the first stage is greater than a first preset threshold, the domain classification loss of the sample set in the second stage is greater than a second preset threshold and less than the first preset threshold, and the domain classification loss of the sample set in the third stage is less than the second preset threshold.

[0009] In some embodiments of the first aspect of the present invention, the target detection module further includes: a data augmentation module, the data augmentation module including a physically guided generative adversarial network; the forward propagation includes the physically guided generative adversarial network performing data augmentation on the sample set.

[0010] In some embodiments of the first aspect of this invention, the physics-guided generative adversarial network is a generative adversarial network guided by an atmospheric scattering model and / or a rain stripe model.

[0011] In some embodiments of the first aspect of the present invention, the target detection model includes a feature extraction and fusion module, and the training process of the target detection model further includes: forward propagation and backward propagation based on a sample set of a new domain, the new domain corresponding to a third weather condition, and the total loss further includes feature distillation loss, the feature distillation loss being obtained based on a first feature of the current sample set and a first feature of the historical sample set in the memory replay library, the first feature being obtained by the feature extraction and fusion module.

[0012] In some embodiments of the first aspect of the present invention, the forward propagation includes the processing of the sample set by the feature extraction and fusion module, wherein the processing of the feature extraction and fusion module includes: aligning image features and point cloud features using a cross-modal attention mechanism.

[0013] According to a second aspect disclosed in this invention, a domain-adaptive target detection apparatus for implementing the above-described domain-adaptive target detection method is provided, comprising:

[0014] A training unit is used to train an object detection model using a sample set. The object detection model includes a dynamic domain adaptation module, which includes an image-level domain classifier with a first dynamic adversarial gradient inversion layer and an object-level domain classifier with a second dynamic adversarial gradient inversion layer. The sample set includes samples from the target domain and samples from the source domain. The target domain corresponds to a first weather condition, and the source domain corresponds to a second weather condition. The training process of the object detection model includes forward propagation and backward propagation based on the sample set. The forward propagation includes calculating the total loss of the object detection model, which includes an image-level domain classification loss and an object-level domain classification loss. In the backward propagation, the gradient inversion strength of the first dynamic adversarial gradient inversion layer is dynamically adjusted according to the image-level domain classification loss. In the backward propagation, the gradient inversion strength of the second dynamic adversarial gradient inversion layer is dynamically adjusted according to the object-level domain classification loss.

[0015] The detection unit is used to acquire images captured by a camera mounted on the vehicle and point clouds captured by a radar mounted on the vehicle under the first weather conditions, and to obtain the target detection result of the vehicle based on the images and point clouds using the target detection model.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the method described above.

[0017] According to a fourth aspect of the present invention, a computer-readable storage medium storing a program is provided, the program including instructions that, when executed by a processor, cause the processor to perform the method described above.

[0018] Compared with existing technologies, this domain-adaptive target detection method, as well as its apparatus, equipment, and medium, can improve the target detection accuracy under adverse weather conditions, reduce the target false detection rate and target missed detection rate under adverse weather conditions, and thus improve the safety and reliability of autonomous driving systems. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments disclosed in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the system architecture applicable to the embodiments of the present invention;

[0021] Figure 2This is a flowchart illustrating the domain-adaptive target detection method provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the hierarchical structure of the target detection model involved in the embodiments of the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the domain-adaptive target detection device provided in an embodiment of the present invention;

[0024] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0026] The technical solutions of the disclosed embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments disclosed in the present invention, and not all of them. Based on the embodiments disclosed in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0027] The terminology used in the disclosed embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. The singular forms “a,” “the,” and “the” as used in the disclosed embodiments and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] As mentioned earlier, in practical applications, autonomous vehicles will inevitably encounter various severe weather conditions such as rain, fog, snow, and sandstorms. The detection performance of models trained on conventional weather data will decrease significantly under severe weather conditions, leading to problems such as increased false detection rate, increased false detection rate, and reduced positioning accuracy in the target detection of autonomous driving systems, which will affect the safety and reliability of autonomous driving systems.

[0030] In view of this, the present invention discloses a target detection method, apparatus, electronic device and storage medium based on domain adaptation, which minimizes the differences between different domains through image-level and target-level adaptation, and introduces a new adversarial gradient inversion layer to improve the performance of the target detection model on challenging samples, thereby solving the aforementioned problems, improving the target detection accuracy under adverse weather conditions, reducing the target false detection rate and target missed detection rate under adverse weather conditions, and thus improving the safety and reliability of autonomous driving systems.

[0031] Figure 1 A schematic diagram of the structure of an autonomous driving system applicable to an embodiment of the present invention is shown. See also Figure 1 An autonomous driving system may include: electronic devices and peripheral sensor components connected to the electronic devices, including cameras and radar.

[0032] The camera can be used to capture images of the vehicle's environment. In specific applications, the camera can be implemented as, but is not limited to, a single camera, a multi-view camera array, or others. The embodiments disclosed in this invention do not limit the specific deployment method or implementation form of the camera.

[0033] Radar can be used to collect point cloud data of the vehicle's environment. The radar can be, but is not limited to, lidar, millimeter-wave radar, or other types, and can be implemented as a single radar or a radar array. The embodiments disclosed in this invention do not limit the specific type, deployment method, or implementation form of the radar.

[0034] In specific applications, the systems applicable to the embodiments disclosed in this invention can be applied to, but are not limited to, intelligent control of multiple wheeled mobile robots, vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, industrial automation equipment, etc. Vehicles can be, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid vehicles), and special vehicles (e.g., garbage trucks, water trucks, etc.). In other words, the "vehicle" in the embodiments of this invention is equivalent to the aforementioned various devices.

[0035] The embodiments disclosed in this invention can be applied to scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. They are applicable to various aspects including passenger travel, public transportation, logistics distribution, unmanned transportation, last-mile delivery, automated agricultural operations, and automated sanitation. Of course, the embodiments of this invention can also be applied to any other intelligent control scenarios involving equipment such as vehicles. This invention does not limit the application scenarios and applicable fields of the disclosed embodiments.

[0036] The systems applicable to the embodiments disclosed in this invention can be, but are not limited to, autonomous driving systems, intelligent driver assistance systems, traffic jam assist systems, etc. See also Figure 1 The system disclosed in this invention can be installed in a vehicle and used, but is not limited to, an intelligent driver assistance system, a traffic jam assist system, or an autonomous driving system. Furthermore, those skilled in the art should understand that the systems applicable to the embodiments disclosed in this invention are not limited to... Figure 1 The architecture shown is not limited to the above-mentioned application scenarios.

[0037] like Figure 2 The diagram shows a flowchart of a domain-adaptive target detection method provided by an embodiment of the present invention. This target detection method can be implemented through a target detection model, which includes a dynamic domain adaptation module. The dynamic domain adaptation module includes an image-level domain classifier with a first dynamic adversarial gradient reversal layer (AdvGRL) and an object-level domain classifier with a second dynamic adversarial gradient reversal layer.

[0038] See Figure 2 The domain-adaptive target detection method of this invention includes the following steps:

[0039] Step 201: Train the target detection model using a sample set. The sample set includes samples from the target domain and samples from the source domain. The target domain corresponds to the first weather condition, and the source domain corresponds to the second weather condition. The training process of the target detection model includes: forward propagation and backward propagation based on the sample set. Forward propagation includes calculating the total loss of the target detection model. The total loss includes image-level domain classification loss and object-level domain classification loss. In backward propagation, the gradient reversal strength of the first dynamic adversarial gradient reversal layer is dynamically adjusted according to the image-level domain classification loss. In backward propagation, the gradient reversal strength of the second dynamic adversarial gradient reversal layer is dynamically adjusted according to the object-level domain classification loss.

[0040] Step 202: Under the first weather conditions, acquire images collected by the camera mounted on the vehicle and point clouds collected by the radar mounted on the vehicle, and use a target detection model to obtain the target detection results of the vehicle based on the images and point clouds.

[0041] The first weather condition can be, but is not limited to, severe weather conditions such as rain or fog, while the second weather condition can be sunny or cloudy with good visibility. Source domain samples are labeled samples, while target domain samples are unlabeled samples. Collaborative optimization of the source and target domains can reduce labeling dependencies.

[0042] like Figure 3 The diagram shown is a structural schematic of the target detection model involved in an embodiment of the present invention. See also... Figure 3 The target detection model in this embodiment of the invention may include an input module, a data augmentation module, a feature extraction and fusion module, a dynamic domain adaptation module, and a detection optimization module connected in sequence. In the dynamic domain adaptation module, adding dynamic AdvGRL to the image-level domain classifier can achieve global feature alignment; adding dynamic AdvGRL to the object-level domain classifier can enable depth alignment of local features of the target detection model, significantly improving appearance invariance and size invariance.

[0043] In step 201, during the training phase of the target detection model, the sample set enters the target detection model through the input module, and then passes through the data augmentation module, feature extraction and fusion module, dynamic domain adaptation module, and detection optimization module to complete forward propagation. After calculating the total loss, it passes through the detection optimization module, feature extraction and fusion module, and input module to complete backpropagation, thereby updating the parameters of the target detection model.

[0044] In step 202, the image and point cloud are input into the target detection model via the input module, and then processed by the feature extraction and fusion module and the detection optimization module to obtain the target detection result. For example, the target detection result may include information such as the location and category of objects around the vehicle. The location can be represented by a bounding box, and the category can be preset, including but not limited to pedestrians, vehicles, traffic signs, etc.

[0045] In step 201, the gradient reversal strength of the first dynamic adversarial gradient reversal layer and the gradient reversal strength of the first dynamic adversarial gradient reversal layer are dynamically adjusted in the following way: when the domain classification loss is less than the preset domain classification loss threshold, the gradient reversal strength is set to the smaller value between the quotient obtained by dividing the preset fixed value by the domain classification loss and the preset upper limit of the gradient reversal strength; and when the domain classification loss is greater than or equal to the domain classification loss threshold, the gradient reversal strength is set to the preset fixed value.

[0046] Here, the domain classification loss can be either the image-level domain classification loss or the object-level domain classification loss mentioned above.

[0047] Specifically, the gradient reversal strength satisfies the following expression:

[0048] ,

[0049] In the formula, λ adv L represents the gradient reversal strength. c Represents the domain classification loss, L c The size of L reflects the performance of the target detection model on hard examples (e.g., small targets in dense fog). c A smaller value indicates better performance of the object detection model. λ0 is a preset fixed value, α is the domain classification loss threshold, and β is the upper limit of gradient inversion strength. λ0, α, and β are all preset parameters; for example, λ0=1.0, α=0.63, and β=30.

[0050] For the gradient inversion strength of the first dynamic adversarial gradient inversion layer, L c L represents the image-level domain classification loss, and for the gradient reversal strength of the second dynamic adversarial gradient reversal layer, L c This represents the object-level domain classification loss.

[0051] L c When α < α, the domain classification loss is small. A small loss indicates that the object detection model performs well on difficult features, demonstrating sufficient ability to handle additional challenges. In this case, the gradient reversal strength can be increased; that is, the gradient reversal strength can be set to λ0 / L. c Smaller values ​​between β and β are used to enhance domain alignment training, forcing the object detection model to learn domain-invariant features.

[0052] L c ≥α indicates a large domain classification loss, suggesting that the object detection model is still learning basic capabilities on hard example features. Therefore, the gradient reversal strength can be kept at a preset fixed value λ0, that is, the basic gradient reversal strength can be maintained, while the domain alignment strength can be reduced to avoid adversarial training interference with the main task optimization.

[0053] The domain classification loss threshold α can be adaptively adjusted based on the exponential moving average (EMA). Specifically, the loss sequence can be smoothed using EMA to capture the long-term trend of the loss distribution, and the domain classification loss threshold α can be dynamically adjusted accordingly.

[0054] The aforementioned dynamic adjustment strategy for gradient reversal strength is achieved through real-time feedback of domain classification loss. When performing well on difficult examples, domain alignment is strengthened, while when performing poorly on difficult examples, the focus is on basic capabilities. By limiting the upper limit of adversarial strength with β, gradient explosion is avoided, thereby improving the generalization ability of the object detection model. This solves the problem of "when to strengthen alignment" in domain adaptation, significantly enhancing the generalization ability of the object detection model in complex real-world scenarios. It is especially suitable for applications such as autonomous driving that require handling domain offset.

[0055] In embodiments of the present invention, by using dynamic AdvGRL to drive image-level and object-level domain classifiers and training the target detection model in a way that confuses the differences in domain features, the detection accuracy of the target detection model on difficult cases such as small targets in dense fog can be improved.

[0056] In step 201, the object detection model can be trained using a phased optimization approach, focusing on both easy and difficult samples. Specifically, the training process of the object detection model can include a first stage, a second stage, and a third stage executed sequentially. In the first stage, the domain classification loss of the sample set is greater than a first preset threshold; in the second stage, the domain classification loss of the sample set is greater than a second preset threshold but less than the first preset threshold; and in the third stage, the domain classification loss of the sample set is less than the second preset threshold. This phased training strategy, which initially focuses on easy samples for stable convergence and then strengthens optimization of difficult examples in the later stages, can better balance accuracy and efficiency.

[0057] For example, the first preset threshold can be set to 0.7, 0.8, or other values, and the second preset threshold can be set to 0.4, 0.5, or other values. Specifically, in the first stage, 0-20k iterations, only easy samples (i.e., L...) are optimized. c The first stage, with samples >0.7, represents a stable initial training phase; the second stage, involving 20k-50k iterations, introduces medium-sized samples (i.e., 0.4). <L c For samples with a complexity <0.7, the complexity is gradually increased. In the third stage, with 50k+ iterations, the complexity is comprehensively optimized for difficult samples (i.e., L...). c (For samples with a value <0.4), improve the final accuracy.

[0058] Verification has shown that training the target detection model using the aforementioned phased optimization method for difficult samples can improve the accuracy of difficult sample detection by about 4.6% and accelerate the training convergence speed by about 14%.

[0059] Traditional domain adaptation uses a constant gradient reversal strength, which cannot dynamically optimize hard examples such as targets obscured by dense fog. This limits the detection accuracy of the target detection model and also suffers from slow convergence speed, low stability, and insufficient optimization of hard examples. The target detection model disclosed in this invention uses dynamic AdvGRL in both the image-level domain classifier and the object-level domain classifier. During the training phase of the target detection model, the gradient reversal strength of the dynamic AdvGRL is dynamically adjusted according to the domain classification loss. Simultaneously, a phased training strategy is adopted to gradually optimize easy and difficult samples. The hard example detection accuracy of the target detection model is improved by approximately 4.6%, the average precision (AP) of pedestrian detection is improved from 38.5% to 43.1%, the overall mean average precision (mAP) is improved by approximately 2.5%, and the training convergence speed is accelerated by approximately 14%.

[0060] Furthermore, the data augmentation module includes a physically guided generative adversarial network (GAN), and the forward propagation in step 201 involves using the GAN to augment the sample set. Specifically, before the sample set enters the feature extraction and fusion module, the GAN can be used to perform generative data augmentation on the sample set, generating realistic fog / rain images to expand the sample set and thus improve its data diversity. Therefore, using physical models such as atmospheric scattering and raindrop dynamics in conjunction with a GAN to perform generative augmentation on the sample set can improve domain-adaptive data diversity.

[0061] Based on a sample set containing both target domain and source domain samples, generative augmentation can be performed using a physically guided generative adversarial network to expand the source domain, target domain, and auxiliary domain in the sample set, constructing all-weather samples and improving the diversity of domain-adaptive samples.

[0062] Specifically, a physics-guided generative adversarial network can be, but is not limited to, a generative adversarial network guided by an atmospheric scattering model and / or a rain stripe model.

[0063] The Atmospheric Scattering Model describes the physical scattering of light in a medium. Its core physical scenario is the attenuation and color shift effect of light passing through suspended particulate matter (such as fog and haze).

[0064] In some examples, the expression for the atmospheric scattering model is:

[0065] ,

[0066] In the formula, J(x) represents the clear image, A represents atmospheric light, d(x) represents the depth map, which can be obtained from radar point clouds or through monocular depth estimation, I(x) represents the generated foggy image, and β is the atmospheric scattering coefficient, which is used to control the fog concentration. Increasing β indicates that the fog concentration increases and the visibility decreases. It is usually taken as the color of the brightest area in the image.

[0067] In practical applications, the atmospheric scattering model described above can be used as a hard constraint for physical rules to embed into the generator design or discriminator training, ensuring that the data generated by the physical-guided generative adversarial network conforms to optical laws.

[0068] When it is necessary to generate realistic images under rainy conditions, the rain stripe model can be embedded as a physical rule hard constraint into the generator design or discriminator training. This allows the physics-guided generative adversarial network to use the rain stripe model to synthesize dynamic raindrops and superimpose motion blur effects, thereby generating realistic rainy images.

[0069] Physics-guided GANs enhance data diversity while ensuring that the generated data conforms to the physical consistency of the real world by explicitly embedding physical laws as constraints on the generator.

[0070] In step 201, after the physical-guided generative adversarial network (GAN) augments the sample set, domain randomization can be used to introduce parameter perturbations into the data generated by the physical-guided GAN. This generates diverse samples by randomly sampling dynamic weather parameters, thereby covering a wider range of target domain distributions. For example, in autonomous driving simulation, after generating basic fog effects based on a physical model, parameters such as raindrop density, illumination angle, and camera noise can be further randomized to generate diverse samples.

[0071] Verification showed that after physically-guided generative adversarial networks augmented the sample set, further generation of diverse samples through domain randomization reduced the Frechet Inception Distance (FID) of the generated images by 25%, further improved the mean average precision (mAP) of domain adaptation by 2.5%, and reduced the vehicle false detection rate in dense fog by approximately 18%.

[0072] In step 201, the forward propagation during the training of the target detection model may further include: domain-level metric regularization, which includes: generating samples in the auxiliary domain and adding triplet loss constraints to the total loss, wherein the triplet includes the source domain, the target domain, and the auxiliary domain.

[0073] Specifically, auxiliary domain samples can be generated using a physically-guided generative adversarial network (GAN). Specifically, continuous weather degradation is simulated on images and point clouds to synthesize physically realistic intermediate transition domain samples (such as samples under mixed rain and fog weather), bridging the distributional differences between the source domain (e.g., sunny weather) and the target domain (e.g., dense fog). The auxiliary domain generation step can provide the samples needed for triplet loss.

[0074] The triplet loss constraint includes: using source domain sample features as anchors, auxiliary domain sample features as positive examples, and target domain sample features as negative examples, the triplet loss is calculated based on the triplet loss function. The triplet loss is calculated in each training batch and enters subsequent backpropagation along with the main losses of the object detection model, such as the domain classification loss and detection loss.

[0075] In step 201, during the training of the object detection model, the calculation of the total loss in the forward propagation includes: calculating the detection loss. For source domain samples, the detection loss can be calculated using the labels of the source domain samples and the object detection prediction results obtained from their forward propagation.

[0076] For target domain samples, the detection loss can be achieved through pre-training with a Masked Autoencoder (MAE) task or iterative optimization using pseudo-labels. Specifically, the detection loss for target domain samples can be the mean squared error (MSE) of the masked region, or it can be calculated from the pseudo-labels of the target domain samples and their target detection prediction results. Here, pseudo-labels can be selected from existing target detection prediction results using a high-confidence screening method.

[0077] The detection optimization module can include a Region Proposal Network (RPN) and a detection head. The PRN can efficiently generate candidate target regions (Region Proposals), which are bounding boxes in the image that may contain objects. The detection head can output the target's category probability and location information through the collaborative work of a classification branch and a regression branch. In the training phase of step 201, the input to the detection head is the first feature processed by the dynamic domain adaptation module. In the detection phase of step 202, the input to the detection head is the first feature output by the feature extraction and fusion module.

[0078] The training process of the target detection model also includes forward and backward propagation based on the sample set of the new domain, which corresponds to the third weather condition. The total loss may also include feature distillation loss, which can be obtained based on the first feature of the current sample set and the first feature of the historical sample set in the memory playback library. The first feature is obtained by the feature extraction and fusion module.

[0079] Here, the third weather condition also falls under the category of severe weather conditions. In one example, the first weather condition could be dense fog, the second weather condition could be sunny, and the third weather condition could be rainy.

[0080] The expression for calculating characteristic distillation loss is:

[0081] ,

[0082] In the formula, F new F represents the first feature of the current sample set. old L represents the first feature of the historical sample set in the memory replay library. distill This indicates characteristic distillation loss.

[0083] The purpose of using feature distillation loss is to achieve consistency in the feature distribution between the constrained new domain (e.g., rainy day) and the aforementioned target domain (e.g., foggy day).

[0084] The memory replay library stores key samples (such as high-loss samples) from the source domain and the historical target domain, which are randomly replayed when training new tasks.

[0085] Most domain-adaptive object detection methods in related technologies are single-transfer methods without a memory replay mechanism. This leads to catastrophic forgetting of object detection models in scenarios with continuous weather changes, resulting in significant performance degradation in historical target domains such as foggy weather. Therefore, the embodiments disclosed in this invention incorporate training based on a new domain sample set into the object detection model's training process, enabling continuous learning. Elastic feature distillation constrains the consistency of features between the old and new domains, preventing catastrophic forgetting. A memory replay library stores key samples from the historical domain, which are randomly replayed during training of new tasks, maintaining the model's cross-domain generalization ability and balancing cross-domain distribution. Verification shows that by incorporating training based on a new domain sample set into the object detection model's training process, the performance degradation in the old domain decreases from 25% to 5%, while the mAP in the new domain (e.g., rainy weather) improves by approximately 1.8%, and the training data requirement is reduced by 30%.

[0086] As can be seen, the target detection model disclosed in this invention supports cross-domain continuous learning, better adapts to the dynamic changes of complex weather, and avoids performance degradation.

[0087] In some implementations, during the forward propagation of the training process in step 201 and the process of obtaining the target detection result using the target detection model in step 202, the processing performed by the feature extraction and fusion module may include: aligning image features and point cloud features using a cross-modal attention mechanism.

[0088] Specifically, aligning image features and point cloud features using a cross-modal attention mechanism can include: projecting the point cloud onto the image plane to generate a sparse depth map, and calculating image features (Q) and radar features (K,V) through cross-attention. Thus, the first feature output by the feature extraction and fusion module can contain both geometric and semantic information, thereby improving the occluded target detection capability of the target detection model.

[0089] In some examples, the feature extraction and fusion module may include a bimodal feature extraction network and a multimodal fusion submodule. The processing of the feature extraction and fusion module may include: extracting point cloud features and image features through a lightweight backbone network, and then aligning and fusing the point cloud features and image features through a cross-modal attention mechanism via the multimodal fusion submodule to obtain fused features.

[0090] The bimodal feature extraction network includes a point cloud branch and an image branch. After the point cloud is voxelized into a 3D mesh, the point cloud branch of the bimodal feature extraction network processes the data to extract point cloud features, which contain the geometric structural features of the vehicle's surrounding environment. The image acquired by the camera is processed by the image branch of the bimodal feature extraction network to extract image features, which contain the texture and semantic features of the vehicle's surrounding environment. Specifically, the bimodal feature extraction network can be, but is not limited to, ConvNeXt-Tiny, and can efficiently extract both point cloud and image features.

[0091] The multimodal fusion submodule can also be used to process the fused features based on the Transformer-enhanced candidate region generation method to obtain anchor-free candidate regions. The aforementioned first feature can be either the fused feature or the anchor-free candidate region.

[0092] Considering the limited sensing capabilities of a single sensor in adverse weather conditions and the high false alarm rate in occluded scenes, the embodiments disclosed in this invention incorporate a cross-modal attention mechanism into the feature fusion extraction module to align image features with point cloud features. This can reduce the false alarm rate in multimodal scenes from 15% to 9%, reduce the false alarm rate of vehicles in dense fog by 18%, and improve pedestrian AP by 5%, effectively improving the detection accuracy of occluded targets.

[0093] The domain-adaptive target detection method disclosed in this invention can be applied to autonomous driving scenarios, achieving high-precision vehicle and pedestrian detection in low-visibility scenarios such as fog and rain.

[0094] The domain-adaptive target detection method disclosed in this invention can also be applied to traffic monitoring systems to improve the target recognition capability of road cameras in adverse weather conditions and support intelligent traffic management.

[0095] like Figure 4 The diagram shows a schematic representation of the target detection device based on domain adaptation provided in an embodiment of the present invention. See also... Figure 4 The domain-adaptive target detection device 400 may include:

[0096] Training unit 401 is used to train an object detection model using a sample set. The object detection model includes a dynamic domain adaptation module, which includes an image-level domain classifier with a first dynamic adversarial gradient inversion layer and an object-level domain classifier with a second dynamic adversarial gradient inversion layer. The sample set includes samples from the target domain and samples from the source domain. The target domain corresponds to the first weather condition, and the source domain corresponds to the second weather condition. The training process of the object detection model includes forward propagation and backward propagation based on the sample set. Forward propagation includes calculating the total loss of the object detection model, which includes the image-level domain classification loss and the object-level domain classification loss. In backward propagation, the gradient inversion strength of the first dynamic adversarial gradient inversion layer is dynamically adjusted according to the image-level domain classification loss. In backward propagation, the gradient inversion strength of the second dynamic adversarial gradient inversion layer is dynamically adjusted according to the object-level domain classification loss.

[0097] The detection unit 402 is used to acquire images collected by a camera mounted on the vehicle and point clouds collected by a radar mounted on the vehicle under the first weather conditions, and to obtain the target detection results of the vehicle based on the images and point clouds using a target detection model.

[0098] Specifically, the training unit 401 can be used to dynamically adjust the gradient reversal strength of the first dynamic adversarial gradient reversal layer and the gradient reversal strength of the first dynamic adversarial gradient reversal layer in the following ways: when the domain classification loss is less than a preset domain classification loss threshold, the gradient reversal strength is set to the smaller value between the quotient obtained by dividing the preset fixed value by the domain classification loss and the preset upper limit of the gradient reversal strength; and when the domain classification loss is greater than or equal to the domain classification loss threshold, the gradient reversal strength is set to a preset fixed value.

[0099] Specifically, the training unit 401 can be used to sequentially execute the first stage, the second stage, and the third stage to complete the training of the target detection model. The domain classification loss of the sample set in the first stage is greater than the first preset threshold, the domain classification loss of the sample set in the second stage is greater than the second preset threshold and less than the first preset threshold, and the domain classification loss of the sample set in the third stage is less than the second preset threshold.

[0100] Specifically, the training unit 401 can be used to augment the sample set using a physically guided generative adversarial network during the forward propagation of the target detection model training process.

[0101] Specifically, the training unit 401 can be used to perform continuous learning of the target detection model. Continuous learning includes forward propagation and backward propagation based on the sample set of the new domain, which corresponds to the third weather condition. The total loss also includes feature distillation loss, which is obtained based on the first feature of the current sample set and the first feature of the historical sample set in the memory playback library. The first feature is obtained by the feature extraction and fusion module.

[0102] In practical applications, the domain-adaptive target detection device 400 can be implemented through software, hardware, or a combination of both. For example, the domain-adaptive target detection device 400 can be implemented as software running in the electronic device 500 described below.

[0103] The present invention also discloses a computer-readable storage medium storing a computer program thereon, the program including instructions that, when executed by one or more processors of a computing device, perform the steps of the domain-adaptive target detection method described in the foregoing method embodiments.

[0104] The present invention also discloses an electronic device 500, which includes one or more processors 501 and a memory 502 storing one or more programs. The program is executed by the one or more processors 501 to implement the method flow and / or program units corresponding to each unit in the apparatus shown in the above embodiments of the present invention.

[0105] The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. Processor 501 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a user interface on an external input / output device (such as a display device coupled to an interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired.

[0106] Processor 501 may include one or more single-core or multi-core processors. Processor 501 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.).

[0107] Memory 502 is the computer-readable storage medium disclosed in this invention, which can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as those in the embodiments disclosed in this invention. Figure 2 The program instructions / units corresponding to the domain-adaptive target detection method are shown. Processor 501 executes non-transient software programs, instructions, and units stored in memory 502, thereby performing operations such as those described in the above method embodiments. Figure 2 The methods shown correspond to the programs, instructions, and units.

[0108] The electronic device 500 may also include an input device 503 and an output device 504. The processor 501, memory 502, input device 503, and output device 504 may be connected via a bus or other means.

[0109] Input device 503 can receive input numerical or character information, and generate relevant signal inputs related to user settings and function control, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 504 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0110] The aforementioned programs (also known as software, software applications, or code) include machine instructions for a programmable processor and can be implemented using object-oriented programming languages, assembly language, or machine language.

[0111] With the development of time and technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media; they can also be downloaded directly from the network. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0112] The technical solutions disclosed in this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas disclosed in this invention. Furthermore, those skilled in the art will recognize that, based on the ideas disclosed in this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Any modifications or equivalent substitutions made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection method based on domain adaptation, characterized in that, The method is implemented through a target detection model, which includes a dynamic domain adaptation module. The dynamic domain adaptation module includes: an image-level domain classifier with a first dynamic adversarial gradient inversion layer and an object-level domain classifier with a second dynamic adversarial gradient inversion layer. The method includes: A target detection model is trained using a sample set, which includes samples from the target domain and samples from the source domain. The target domain corresponds to a first weather condition, and the source domain corresponds to a second weather condition. The training process of the target detection model includes forward propagation and backward propagation based on the sample set. The forward propagation includes calculating the total loss of the target detection model, which includes image-level domain classification loss and object-level domain classification loss. In the backward propagation, the gradient reversal strength of the first dynamic adversarial gradient reversal layer is dynamically adjusted according to the image-level domain classification loss. In the backward propagation, the gradient reversal strength of the second dynamic adversarial gradient reversal layer is dynamically adjusted according to the object-level domain classification loss. Under the first weather conditions, images captured by the camera mounted on the vehicle and point clouds captured by the radar mounted on the vehicle are acquired. The target detection model is then used to obtain the target detection result of the vehicle based on the images and point clouds.

2. The target detection method based on domain adaptation according to claim 1, characterized in that, The gradient reversal strength of the first dynamic adversarial gradient reversal layer and the gradient reversal strength of the first dynamic adversarial gradient reversal layer are both dynamically adjusted in the following way: when the domain classification loss is less than a preset domain classification loss threshold, the gradient reversal strength is set to the smaller value between the quotient obtained by dividing the preset fixed value by the domain classification loss and the preset upper limit of the gradient reversal strength; and when the domain classification loss is greater than or equal to the domain classification loss threshold, the gradient reversal strength is set to the preset fixed value.

3. The target detection method based on domain adaptation according to claim 1 or 2, characterized in that, The training process of the target detection model includes a first stage, a second stage, and a third stage executed sequentially. The domain classification loss of the sample set in the first stage is greater than a first preset threshold, the domain classification loss of the sample set in the second stage is greater than a second preset threshold and less than the first preset threshold, and the domain classification loss of the sample set in the third stage is less than the second preset threshold.

4. The target detection method based on domain adaptation according to claim 1, characterized in that, The target detection module further includes a data augmentation module, which includes a physically guided generative adversarial network; the forward propagation includes the physically guided generative adversarial network performing data augmentation on the sample set.

5. The target detection method based on domain adaptation according to claim 4, characterized in that, The physics-guided generative adversarial network is a generative adversarial network guided by an atmospheric scattering model and / or a rain stripe model.

6. The target detection method based on domain adaptation according to claim 1, characterized in that, The target detection model includes a feature extraction and fusion module. The training process of the target detection model also includes forward propagation and backward propagation based on a sample set of a new domain, the new domain corresponding to a third weather condition. The total loss also includes feature distillation loss, which is obtained based on the first feature of the current sample set and the first feature of the historical sample set in the memory replay library. The first feature is obtained by the feature extraction and fusion module.

7. The target detection method based on domain adaptation according to claim 1 or 6, characterized in that, The forward propagation includes the processing of the sample set by the feature extraction and fusion module, and the processing of the feature extraction and fusion module includes: aligning image features and point cloud features using a cross-modal attention mechanism.

8. A domain-adaptive target detection apparatus for implementing the domain-adaptive target detection method as described in any one of claims 1 to 7, characterized in that, include: A training unit is used to train an object detection model using a sample set. The object detection model includes a dynamic domain adaptation module, which includes an image-level domain classifier with a first dynamic adversarial gradient inversion layer and an object-level domain classifier with a second dynamic adversarial gradient inversion layer. The sample set includes samples from the target domain and samples from the source domain. The target domain corresponds to a first weather condition, and the source domain corresponds to a second weather condition. The training process of the object detection model includes forward propagation and backward propagation based on the sample set. The forward propagation includes calculating the total loss of the object detection model, which includes an image-level domain classification loss and an object-level domain classification loss. In the backward propagation, the gradient inversion strength of the first dynamic adversarial gradient inversion layer is dynamically adjusted according to the image-level domain classification loss. In the backward propagation, the gradient inversion strength of the second dynamic adversarial gradient inversion layer is dynamically adjusted according to the object-level domain classification loss. The detection unit is used to acquire images captured by a camera mounted on the vehicle and point clouds captured by a radar mounted on the vehicle under the first weather conditions, and to obtain the target detection result of the vehicle based on the images and point clouds using the target detection model.

9. An electronic device, characterized in that, include: A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images