Target detection method, detection system, vehicle and visual environment monitoring device thereof

Abstract

The application discloses a target detection method, a detection system, a vehicle and a visual environment monitoring device thereof, and relates to the technical field of computer vision and artificial intelligence. The method comprises the following steps: extracting first inference features from an input foggy image through a main detection flow; inputting the first inference features into an inference-time fusion adapter in the main detection flow to generate second inference features and a pseudo-transmittance map; taking the pseudo-transmittance map as a physical space guide, and performing cross-attention fusion on the first inference features and the second inference features based on the physical guide through a PI-CAF module in the main detection flow to generate enhanced inference features and output a predicted target detection result. The application can solve the training-inference inconsistency problem in the prior art, and realize efficient and physically interpretable feature recognition based on a physically-perceived, interpretable and spatially-adaptive enhancement mechanism, so that real-time and accurate environment perception can be provided for an intelligent transportation system under adverse weather conditions such as foggy weather.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to a target detection method, a detection system, a vehicle and a visual environment monitoring device thereof. BACKGROUND

[0002] In the closed-loop control loop of autonomous driving and intelligent transportation systems, the environmental perception system plays a key role as a "monitor", and its output is the direct basis for the control strategy formulated by the downstream planning and decision-making modules. However, the performance of existing visual detectors will deteriorate sharply in bad weather such as fog, which is one of the core technical bottlenecks hindering the large-scale application of autonomous driving technology. The visual information degradation caused by fog is strongly related to scene depth, making it particularly difficult to identify small and medium-sized objects at a distance.

[0003] To address this challenge, existing research mainly follows a two-stage method or an end-to-end method. The two-stage method first dehazes the image and then performs detection, and its main defect is that the optimization objectives of image restoration and target detection tasks are inconsistent, resulting in suboptimal problems. The end-to-end method attempts to build a unified robust detector, and one of the most promising ideas is to use clear images as "privileged information" for guidance during training. However, since clear images are not available during real-time inference, directly removing the guidance branch will cause serious training-inference inconsistency problems, resulting in a significant loss in model performance.

[0004] In summary, the existing technology lacks a comprehensive solution that can effectively utilize privileged information while fundamentally solving the training-inference inconsistency problem, especially in terms of accurately improving the detection performance of safety-critical targets such as small and medium-sized objects in foggy scenes. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a target detection method, a detection system, a vehicle and a visual environment monitoring device thereof, which can provide real-time and accurate environmental perception and key environmental monitoring information for autonomous driving and intelligent transportation systems under bad weather conditions such as fog.

[0006] To achieve the above purpose, the first aspect of the present application provides a target detection method, comprising: extracting first inference features from an input foggy image through a main detection flow; inputting the first inference features into an inference-time fusion adapter in the main detection flow to generate second inference features and a pseudo-transmittance map; using the pseudo-transmittance map as a physical space guide, performing physical guide-based cross-attention fusion on the first inference features and the second inference features through a PI-CAF module in the main detection flow to generate enhanced inference features and output a predicted target detection result.

[0007] Another aspect of the present application provides a target detection system, comprising: an auxiliary guidance flow, configured to receive training sunny images in a training stage, and generate real features and predicted transmittance maps as teacher guidance signals; and a main detection flow, configured to receive training foggy images paired with the training sunny images in the training stage, and extract first training features in the training foggy images, to generate training enhanced features based on the teacher guidance signals to guide the first training features, and receive input foggy images in an inference stage, to generate enhanced inference features and output final target detection results.

[0008] Another aspect of the present application provides a visual environment monitoring device for a vehicle, comprising: an image acquisition module, configured to acquire a video stream in front of or around the vehicle as a raw input signal; a core processing unit, configured to perform target detection on the raw input signal according to the target detection method to generate monitoring information of the vehicle; and a data interface and output module, configured to output control instructions of the vehicle according to the monitoring information.

[0009] Another aspect of the present application provides a vehicle, comprising: the target detection system; or the visual environment monitoring device.

[0010] Through the above technical solutions, the present application can provide real-time and accurate environment perception for autonomous driving and intelligent transportation systems in adverse weather conditions such as foggy days, based on a physical perception, interpretable, and spatially adaptive enhancement mechanism.

[0011] In addition, the present application can solve the training-inference inconsistency problem in the prior art, realize physically interpretable and efficient features, and at the same time, the present application also adopts an innovative "teaching method" teaching paradigm to realize deep knowledge transfer, accurately solve key problems, and significantly improve the performance of small object detection. The present application can provide key environment monitoring information for intelligent control systems (such as planning and decision control systems of autonomous vehicles), provide high-reliability monitoring input for downstream control systems, and ensure system safety from the source.

[0012] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings:

[0014] Figure 1 A flowchart of a target detection method according to the present application is shown.

[0015] Figure 2 A detailed diagram of the inference-time fusion adapter according to the present application is shown;

[0016] Figure 3 A detailed diagram of the PI-CAF module according to the present application is shown;

[0017] Figure 4 A flowchart showing the specific process of the imitation learning training according to the present application is shown;

[0018] Figure 5 A diagram showing the overall framework of the target detection model according to the present application is shown;

[0019] Figure 6 A diagram showing the main training loss of the composite loss function according to the present application is shown;

[0020] Figure 7 A diagram showing the structure of the target detection system according to the present application is shown;

[0021] Figure 8 A diagram showing the structure of the visual environment monitoring device for vehicles according to the present application is shown;

[0022] Figures 9a-9b A diagram showing the effect comparison of the improved model and the baseline model of the target detection method according to the embodiments of the present application in scene one is shown, wherein, Figure 9a is the improved model, Figure 9b is the baseline model;

[0023] Figures 10a-10b A diagram showing the effect comparison of the improved model and the baseline model of the target detection method according to the embodiments of the present application in scene two is shown, wherein, Figure 10a is the improved model, Figure 10b is the baseline model;

[0024] Figures 11a-11f A diagram showing the visualization analysis of the model internal feature map and weight map of the target detection method according to the embodiments of the present application is shown, wherein, Figure 11a is the final detection result, Figure 11b shows the physical weight map generated by the inference-time fusion adapter according to the atmospheric scattering model principle, Figure 11c is the pseudo-sunny feature intensity map output by the inference-time fusion adapter, Figure 11d is the reference benchmark generated by the auxiliary guidance flow from the corresponding sunny image, Figure 11e is the original foggy feature map before processing by the PI-CAF module, Figure 11f is the feature map after fusion processing by the PI-CAF module. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. It should be understood that the specific implementation described herein is only used to explain and describe the embodiments of the present application, and should not be used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, if the embodiments of the present application involve descriptions such as “first”, “second”, etc., the descriptions of “first”, “second”, etc. are only for description purposes, and should not be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by “first” and “second” can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is also not within the scope of protection claimed by the present application.

[0027] The applicant first briefly describes the technical background and prior art related to the present application.

[0028] 1) Perception challenges in adverse weather and physical principles

[0029] The applicant believes that fog is not simply image noise, but a complex physical phenomenon that can be described by an atmospheric scattering model. For example, by constructing a certain model, the phenomenon of image contrast reduction and color distortion caused by fog can be simulated, and the strong correlation between this degradation and scene depth needs to be considered, so that objects at a long distance and small size are particularly difficult to identify, and these objects are often the key targets that need to be warned and responded to in a traffic scene.

[0030] 2) Technical routes to address the challenge and their limitations

[0031] To address this challenge, existing research mainly follows two technical routes: “two-stage method” and “end-to-end robust detector”.

[0032] Two-stage methods: This approach decomposes the problem into two independent tasks, image defogging and subsequent detection. The main drawback is the existence of a "suboptimal problem" because the goal of image restoration (such as pursuing pixel-level fidelity) is not always consistent with the high-level semantic features that the target detection task focuses on.

[0033] End-to-end robust detector: This approach aims to design a single model that can directly process foggy images. Among them, supervised learning using pairs of clear-fog images is proven to be an effective approach.

[0034] 3) Opportunities and core contradictions of "privileged information" guided learning

[0035] In the end-to-end method, a highly potential idea is to use clear sunny images as "privileged information" during training to guide the model's learning through "teacher-student learning" or knowledge distillation paradigm. However, this advanced idea also introduces its own unique core challenge: in real inference (deployment) scenarios, clear images as the teacher guidance source are not available. If the teacher branch is simply removed during inference, it will result in a significant difference between the network architecture and data flow during training and inference, i.e., the "training-inference inconsistency" problem, which will severely damage the model's performance.

[0036] 4) Baseline detector selection and specific technical challenges

[0037] In addition, the applicant also found that in order to achieve real-time detection, the current single-stage detector represented by the YOLO series is constantly breaking industry standards in speed and accuracy. However, DETR (Detection Transformer) leads the paradigm innovation, which regards target detection as a direct set prediction problem, thus abandoning complex post-processing components such as NMS (Non-Maximum Suppression). RT-DETR (Real-Time Detection Transformer) is the epitome of this direction, which through the design of an efficient hybrid encoder and an IoU (Intersection over Union) aware query selection mechanism, has achieved an excellent balance between speed and accuracy, and is an ideal cornerstone for building the next generation of robust detection systems. However, even with an advanced architecture, the challenge of "accurately detecting small-sized objects in adverse weather" remains a common challenge for the entire field.

[0038] In summary, the prior art lacks a comprehensive solution that requires a new teacher-student learning framework on top of an advanced detection framework (such as RT-DETR) to specifically address the "training-inference inconsistency" problem and utilize physical models in a physically interpretable and final detection task-serving manner to accurately improve the detection performance of small and medium-sized safety-critical targets in foggy scenes.

[0039] To this end, the target detection method 100 provided by the embodiments of the present application can include steps S110-S130, as shown in the flowchart. Figure 1

[0040] Step S110 extracts first inference features from the input foggy image through the main detection flow.

[0041] The target detection method 100 provided by the embodiments of the present application performs the inference stage, i.e., uses the main detection flow (which can be understood as a student) to detect a single input foggy image. The main detection flow can use the RT-DETR-R50 configuration with ResNet (Residual Network)-50 as the backbone network. It contains a Backbone (backbone network) and a Hybrid Encoder (hybrid encoder), which is responsible for extracting first inference features (also known as foggy features), i.e., multi-scale features F foggy .

[0042] However, before performing the target detection task, the target detection method 100 of the present application also needs to include the training stage of the main detection flow. In the training stage, the auxiliary guidance flow (which can be understood as a teacher) also needs to be activated. It can be an efficient U-Net-like network, the core of which is stacked by multiple residual groups (RG, Residual Group), and each residual group contains several residual blocks (RCAB, Residual Channel Attention Block) with channel attention. The auxiliary guidance flow only appears during training and processes clear sky images to provide ideal real features F clear and predicted transmittance maps t pred It can be seen that in the training stage, the main detection flow and the auxiliary guidance flow are used simultaneously to perform imitation learning training using pairs of foggy-sunny images to realize an end-to-end optimization process. In the inference stage, the auxiliary guidance flow is removed and only the main detection flow is used to detect a single input foggy image.

[0043] ​Step S120, input the first inference feature into a fusion adapter in the main detection stream at inference time to generate a second inference feature and a pseudo transmittance map.

[0044] wherein, as shown in Figure 2 , the fusion adapter in the main detection stream at inference time is a lightweight convolutional network embedded in the main detection stream, which is used to learn at the training stage by imitating the guidance signal generated by the auxiliary guidance stream. That is, its only task is to imitate the behavior of the "teacher" auxiliary guidance stream at inference time under the supervision of the adapter imitation loss L adapter , so as to independently generate the second inference feature (also referred to as "pseudo" sunny day feature) and the "pseudo" transmittance map at inference time. Specifically, its input is the foggy day feature extracted by the main detection stream, i.e. the first inference feature, and its output is the predicted pseudo sunny day feature and the pseudo transmittance map.

[0045] Step S130, taking the pseudo transmittance map as a physical space guide, performing physical guide-based cross-attention fusion on the first inference feature and the second inference feature through a PI-CAF (Physical Information-Guided Cross-Attention Fusion) module in the main detection stream to generate an enhanced inference feature and output a predicted target detection result.

[0046] Specifically, the PI-CAF module is used to fuse the foggy day feature with the guidance signal to generate an enhanced inference feature for final detection. In an embodiment, as shown in Figure 3 , the PI-CAF module can perform a physical guide-based cross-attention fusion process in the following steps:

[0047] 1) Prepare input: receive the first inference feature F foggy , and the guidance information, including the second inference feature F guide (the sunny day feature F clear at training time and the pseudo sunny day feature F pseudo_clear at inference time) and the pseudo transmittance map t guide (the predicted transmittance map t pred at training time and the pseudo transmittance map t pseudo_pred at inference time). Then, according to the transmittance of each point on the pseudo transmittance map t guide , determine the physical unreliability weight of each pointW unreliability =1- t guide .

[0048] 2) Standard cross-attention: Perform multi-head cross-attention, letting F foggy query and aggregate relevant information from F guide and output preliminary fused features F attended . Specifically, take the first reasoning feature F foggy as Query, and the second reasoning feature as Key and Value, perform cross-attention operation, and generate attention-enhanced reasoning feature F attended .

[0049] 3) Physically-guided weighted fusion: This is the key step. The theoretical basis of this module comes from the atmospheric scattering model (ASM): I ( x )= J ( x ) t ( x )+A(1- t ( x ))· W unreliability =1- t ( x ) can be defined as a physical unreliability weight. This weight quantitatively describes the degree to which the original scene information is obscured by atmospheric light at each spatial position x . This weight map provides an interpretable physical prior for the strength of feature fusion. Then, based on the physical unreliability weight W unreliability , dynamically mix the first reasoning feature F foggy and the attention-enhanced reasoning feature F attended to determine the enhanced fusion feature at each point:

[0050] F fused = F foggy ·(1- W unreliability )+ F attended · W unreliability .

[0051] 4) Output projection: the enhanced fusion features after dynamic mixing are integrated by a linear layer and output.

[0052] The above operation has the meaning that it uses physical priors to focus the enhanced resources on the most severely degraded regions of the signal, while also being a risk control mechanism that trusts the original features more in regions where the original information is clear, avoiding the introduction of potential artifacts.

[0053] It can be seen that based on the above steps S110-S130, the execution steps of the present application in the inference stage are shown, including: first, the input single foggy image is processed by the main detection flow to generate foggy features F foggy The auxiliary guidance flow does not participate at all; then the trained inference-time fusion adapter takes over the work to process F foggy and independently generate high-quality pseudo-sunny features F pseudo_clear and pseudo-transmittance maps t pseudo_pred Finally, using the guidance information generated by the student itself, the PI-CAF module is used to enhance and dynamically mix F foggy to generate enhanced fusion features and send them to the decoder for final detection.

[0054] In an embodiment, the main detection flow also needs to perform imitation learning training 200 in the training stage, as shown in the flowchart. Figure 4 The specific process of the imitation learning training 200 can include the following steps S210-S260.

[0055] Step S210, obtaining a pair of training foggy image and training sunny image;

[0056] Step S220, processing the training foggy image by the main detection flow to extract first training features;

[0057] Step S230, processing the training sunny image by the auxiliary guidance flow to generate real features and predicted transmittance maps as teacher guidance signals;

[0058] Step S240, inputting the first training features into the inference-time fusion adapter to generate second training features and pseudo-transmittance maps as student outputs;

[0059] Step S250, using the teacher guidance signals to perform cross-attention fusion based on physical guidance on the first training features to generate enhanced training features and output predicted target training results;

[0060] Step S260, a composite loss function including the physical supervision loss, the feature consistency loss and the imitation loss is constructed, and the end-to-end joint optimization of the main detection flow is performed with the minimization of the composite loss function as the optimization objective.

[0061] In addition, in step S250, the obtained enhanced training features also need to be decoded for detection, so as to output the predicted target training results.

[0062] In order to perform the end-to-end joint optimization of the system, the application also designs a composite loss function, in addition to the above-mentioned losses, the composite loss function also needs to include a detection loss, which can be defined as L det For example, the detection loss can directly use the standard detection loss combination defined in RT-DETR, including a variable focus loss (VFL, VarifocalLoss), an L1 loss and a GIoU loss (Generalized IoU Loss), which is the core task loss of the model.

[0063] Specifically, the physical supervision loss, the feature consistency loss and the imitation loss in the composite loss function are determined through the following steps S261-S263 respectively:

[0064] Step S261, the physical supervision loss is determined by the theoretical true transmittance map and the predicted transmittance map.

[0065] In order to ensure that the teacher flow can learn the physically correct transmittance map, the application directly supervises the predicted transmittance map output by the teacher flow t pred The loss supervised is L phy defined as the physical supervision loss.

[0066] The theoretical true transmittance map is the transmittance map between the training foggy image and the training sunny image, which can be determined based on the inverse solution of the atmospheric scattering model. First, the global atmospheric light value A is determined from the training foggy image I foggy by the dark channel prior (DCP) algorithm; then, according to the atmospheric scattering model I ( x )= J ( x ) t ( x )+A(1- t ( x )) , the training foggy image, the training sunny image and the global atmospheric light value are used to inversely solve the theoretical true transmittance map t gt.in, I ( x ) represents training foggy images (foggy images). J ( x ) represents training images on a clear, sunny day (clear images). t ( x () represents transmittance; finally, calculate... t pred and t gt Physical monitoring loss between them.

[0067] Step S262, the imitation loss is determined by the real features and the second training features, as well as the predicted transmittance map and the pseudo transmittance map.

[0068] Can be defined L adapter This is a mimicry loss. The sole purpose of this loss term is to penalize the adapter's output (i.e., the second trained feature) during the fusion of the adapter in training inference. F pseudo_clear Pseudotransmittance map t pseudo_pred ) and the output of the teacher flow (i.e., true features) F clear Predicted transmittance map t pred The difference between the two is the loss. This loss is the core of the "guidance," which generates gradients specifically to update the parameters of the fusion adapter during inference, telling it how to adjust itself to make predictions that are closer to the "teacher."

[0069] Step S263, the feature consistency loss is determined by enhancing the training features and the true features.

[0070] Can be defined L feat Feature consistency loss, as a regularization constraint, guides the fused augmented training features to perceptually resemble the ideal, clear real features. F clear To maintain similarity, a pre-trained VGG16 network with frozen parameters on ImageNet can be used as a fixed "feature arbiter." This loss does not directly compare pixel values, but rather compares the enhanced trained features with the true features of a sunny day. F clear The difference between activation maps output from one or more intermediate convolutional layers in a VGG network is used to measure the similarity between the two in terms of high-level visual features such as content, structure, and texture. Specifically, high-level structural and textural similarity is measured by calculating the distance between activation maps output from multiple intermediate convolutional layers in a VGG16 network.

[0071] Reference is additionally made to Figure 5 The training process can be performed in the training stage by the following steps:

[0072] 1) The main detection stream (student) processes the training foggy images to generate foggy features, i.e. the first training features F foggy ;

[0073] 2) The auxiliary guidance stream (teacher) processes the training sunny images to generate the real features of sunny F clear and the predicted transmittance map t pred ;

[0074] 3) The inference-time fusion adapter in the main detection stream (student) processes the foggy features to generate the predicted pseudo-sunny features, i.e. the second training features F foggy and the pseudo-transmittance map F pseudo_clear ; t pseudo_pred ;

[0075] 4) Using the real F clear and t pred generated by the teacher in step 2), the F foggy is enhanced by the PI-CAF module, and the physical supervision loss for optimizing the main detector is calculated L phy ;

[0076] 5) Comparing the output imitated by the student in step 3) with the real output of the teacher in step 2), the adapter imitation loss L adapter is calculated for optimizing the inference-time fusion adapter.

[0077] In addition, the expression of the compound loss function L total can be:

[0078] L total = L det + λ feat · L feat + λ phy · L phy + λ adapter · L adapter

[0079] wherein, λ feat , λ phy , λ adapter are the weight coefficients corresponding to each loss respectively.

[0080] Referring to Figure 6 , a schematic diagram of the main training losses of the composite loss function is shown.

[0081] Compared with the prior art, the present application has the following remarkable beneficial effects:

[0082] 1) fundamentally solve the training-inference inconsistency problem and realize high feasibility deployment. The present application introduces a unique inference-time fusion adapter module for imitative learning, which fundamentally solves the deployment dilemma when using "privileged information" (sunny day images) for guided learning. This design enables the powerful teacher-student learning paradigm to run in an efficient and self-consistent single-stream architecture during inference, with very high engineering deployment feasibility.

[0083] 2) realize physically interpretable efficient feature enhancement. The core innovation PI-CAF module of the present application is not a black box. It deeply integrates the atmospheric scattering physical model with the cross-attention mechanism, and intelligently and spatially adaptively performs "targeted repair" on the damaged features through a physically interpretable "unreliability weight map". Visualization analysis clearly shows this interpretable repair process, providing reliability assurance for safety-critical applications.

[0084] 3) innovative "teaching method" teaching paradigm, realizing deep knowledge transfer. Unlike traditional knowledge distillation, which focuses on "guiding answers", the present application is a more advanced "teaching method" paradigm. Instead of directly imparting results, the teacher provides high-quality intermediate components (ideal sunny day features and physical transmittance maps) necessary for executing the enhanced "method", and the student's goal is to learn how to independently generate these key intermediate components. This mode realizes deeper and generalizable knowledge transfer.

[0085] 4) precisely tackle key problems and significantly improve small and medium-sized object detection performance. The present application ensures that all improvements are most efficiently translated into detection accuracy improvements. Experimental quantitative data shows that the present framework precisely solves the most serious pain point of feature degradation of small and medium-sized objects in foggy environments, and the performance gain is mainly reflected in small and medium-sized objects, with the detection accuracy of small objects improving by up to 78.65%. This has crucial practical significance for ensuring the safety of autonomous driving (such as early detection of pedestrians or motorcycles in the distance).

[0086] 5) Provide high-reliability monitoring input for downstream control systems, ensuring system safety from the source. This application essentially provides a "eye" for control systems such as autonomous driving that can still work stably in bad weather. By continuously outputting stable, accurate, and non-missing key target monitoring information to the planning, decision-making, and control modules, it directly ensures the effectiveness and safety of the downstream control strategy, and solves the systemic risk caused by environmental perception failure from the source of the entire control link.

[0087] On the other hand, the application also provides a target detection system 300 configured to perform a target detection task, for example, a robust visual environment monitoring device for a control system. As shown in the structural schematic diagram of Figure 7 The target detection system 300 can include a main detection flow 310, and additionally include an auxiliary guidance flow 320 in the training phase, and the auxiliary guidance flow 320 is not activated in the inference phase.

[0088] The main detection flow 310 can be used to: in the training phase, receive a training fog image paired with a training sunny image, and extract a first training feature in the training fog image, to generate a training enhanced feature based on a teacher guidance signal to guide the first training feature; in the inference phase, receive an input fog image, to generate an enhanced inference feature and output a final target detection result.

[0089] Specifically, the main detection flow 310 can be understood as a student for detecting a single input fog image. The main detection flow 310 can adopt an RT-DETR-R50 configuration with ResNet-50 as the backbone network. It contains Backbone and Hybrid Encoder, which is responsible for extracting the first training feature (also known as fog feature) from the input fog image, that is, multi-scale feature F foggy It can be seen that the main detection flow 310 needs to independently receive the fog image in the inference phase, extract the fog feature, and perform self-guided feature enhancement to generate the final detection result.

[0090] The auxiliary guidance flow 320 is only activated in the training phase, and is used to receive a training sunny image paired with the fog image in the training phase, and generate a high-quality sunny real feature F clear and a predicted transmittance map t predAs a teacher guidance signal. At the same time, the guidance signal is sent to the main detection stream 310 for guidance to enhance the fog day features to generate training enhanced features. Specifically, the auxiliary guidance stream 320 can be understood as a teacher, which can be an efficient U-Net-like network, the core of which is stacked by multiple residual groups, and each residual group contains several residual blocks with channel attention.

[0091] It can be seen that the main detection stream 310 and the auxiliary guidance stream 320 need to be used at the same time in the training stage, that is, the paired fog-sunny images are used for imitation learning training to realize an end-to-end optimization process. And then in the inference stage, the auxiliary guidance stream 320 needs to be removed, and only the main detection stream 310 is used to detect the input single fog image.

[0092] In an embodiment, the main detection stream 310 can further include: a feature extraction module 311 for extracting first inference features from the fog image; an inference-time fusion adapter 312 for generating second inference features and a pseudo-transmittance map according to the first inference features; a PI-CAF module 313 for using the pseudo-transmittance map as a physical space guide to perform physical guide-based cross-attention fusion on the first inference features and the second inference features to generate enhanced inference features; and a decoding and prediction module 314 for outputting a predicted target detection result according to the enhanced inference features.

[0093] The inference-time fusion adapter 312 is a lightweight convolutional network embedded in the main detection stream 310. Its only task is to imitate the behavior of the “teacher” auxiliary guidance stream 320 under the supervision of the adapter imitation loss during training, so as to independently generate pseudo-sunny features and pseudo-transmittance maps during inference. The input of the adapter is the fog features extracted by the main detection stream, and the output is the predicted “pseudo” sunny features and “pseudo” transmittance maps; the adapter learns by imitating the guidance signal generated by the auxiliary guidance stream during the training stage.

[0094] Specifically, the PI-CAF module 313 is used to fuse the fog features with the guidance signal to generate enhanced features for final detection. The fusion process of the PI-CAF module 131 is guided by a physical unreliability weight (PIW) calculated based on a physical model. W unreliability The theoretical basis of this module comes from the atmospheric scattering model: I ( x )= J ( x ) t ( x )+ A (1- t ( x ))。

[0095] Based on this, the W unreliability =1- t x is defined as a physical unreliability weight. The weight quantitatively describes the degree to which the original scene information is covered by the atmospheric light at each spatial position x The weight map provides an interpretable physical prior for the strength of feature fusion, which is used to dynamically adjust the proportion of the original foggy feature and the guidance information in the fusion process. In the area where the fog concentration is high, the weight of the guidance information is increased.

[0096] On the other hand, the application also provides a visual environment monitoring device 400 for a vehicle, as shown in the structural schematic diagram, the visual environment monitoring device 400 can include: Figure 8

[0097] The image acquisition module 410 is used to acquire the video stream in front of or around the vehicle as the original input signal. The image acquisition module 410 is one or more vehicle-mounted high-definition cameras, for example, which is responsible for capturing the video stream in front of or around the vehicle in real time as the original input signal of the monitoring device.

[0098] The core processing unit 420 is used to perform target detection on the original input signal according to the target detection method 100 described above to generate monitoring information of the vehicle. The core processing unit 420 is a vehicle-mounted intelligent computing platform, such as the NVIDIA Jetson series, for example. The function description is to load and run the robust target detection model trained and solidified with the method of the application; receive real-time image data from the image acquisition module; perform forward inference of the model to output real-time monitoring information such as the class and position of the target in the environment.

[0099] The data interface and output module 430 is used to output control instructions for the vehicle according to the monitoring information. Specifically, the data interface and output module 430 can encapsulate and broadcast the generated monitoring results (such as the coordinates, class, and confidence of the target) through the vehicle-mounted network (such as CAN bus or vehicle-mounted Ethernet) for subscription and use by the downstream control systems of the vehicle such as path planning, decision-making, or emergency braking.

[0100] In addition, in the deployment stage, the deployment process of each module or model in the visual environment monitoring device 400 can be: using the inference optimization engine such as NVIDIA TensorRT to optimize the trained deep learning model, including quantization, operator fusion, etc. Operation to generate an inference engine that can run at high speed on the target hardware platform, and finally deploy it in the operating system of the core processing unit.

[0101] ​​On the other hand, this application also provides a vehicle that may include the target detection system described above and / or the visual environment monitoring device described above.

[0102] The beneficial effects of the target detection system, visual environment monitoring device, and vehicle provided in this application can be referred to the above description of the target detection method, and will not be repeated here.

[0103] To further illustrate the substantive features of this application, the following provides experimental verification of the proposed solution and comparative analysis with existing technologies, thereby demonstrating the beneficial technical effects of this application.

[0104] I. Experimental Setup

[0105] 1) Dataset: The experimental data comes from the publicly available Cityscapes dataset and its corresponding FoggyCityscapes dataset. 2965 pixel-aligned image pairs were selected for training, and 490 pairs were used for validation.

[0106] 2) Evaluation Categories: To focus on autonomous driving scenarios, experiments and evaluations are conducted on a subset of five core categories: person, car, bus, rider, and traffic light.

[0107] 3) Baseline model: The same RT-DETR-R50 model was selected as the baseline for fair comparison.

[0108] 4) Evaluation metrics: The standard COCO evaluation metrics are used, including mean average precision (mAP), average precision for small objects (AP_S), average precision for medium objects (AP_M), and average precision for large objects (AP_L), and the corresponding average recall (AR).

[0109] II. Experimental Results

[0110] 1) Quantitative analysis: The experimental results are shown in Table 1.

[0111] Table 1: Comparison of metrics between the improved model and the baseline model

[0112]

[0113] Referring to the data shown in Table 1, compared with the baseline model in the prior art, the PG-DB-RTDETR of the present application is improved from 37.5% to 38.6% in comprehensive accuracy (AP@.50:.95), with an increase of 1.1 percentage points (relative improvement 2.90%), and the performance gain is mainly reflected in the most challenging targets in foggy weather. In terms of small object (AP_S) detection, the accuracy is improved from 5.2% to 9.3%, with an absolute increase of 4.1 percentage points, and the relative performance improvement is 78.65%. In terms of medium object (AP_M) detection, the accuracy is also improved from 32.5% to 35.7%, with an increase of 3.2 percentage points (relative improvement 9.93%). At the same time, the recall rate of small objects (AR_S) is also improved by 30.23%. These data show that the physical guidance fusion mechanism of the present application helps to restore and enhance the medium and high frequency detail information that is easily damaged in foggy weather, and these information is crucial for identifying small and medium-sized objects. In addition, it is worth noting that the large object detection accuracy (AP_L) has a slight decrease (-0.47%). This phenomenon may indicate that the model allocates more network capacity to the feature restoration of more challenging small and medium-sized targets during the optimization process, thereby improving the overall performance while reflecting the targeted optimization of difficult samples.

[0114] In addition, by comparing the detection results of the improved model and the baseline model in typical scenes (as shown in Figs. 10a-10b), the improvement in model performance can be intuitively evaluated. Figures 9a-9b

[0115] In scene one (the improved model is Figure 9a , and the baseline model is Figure 9b ), the improved model using the scheme of the present application shows higher detection stability and detection ability. First, the recognition confidence of the model for multiple long-range targets is improved, for example, the confidence of the vehicle affected by fog is improved from 0.51 and 0.55 to 0.60 and 0.64, respectively. Second, the model reduces missed detection and detects an additional vehicle (confidence 0.67) in the thick fog area that is missed by the baseline model, making up for the perception blind area.

[0116] In scene two (the improved model is Figure 10a , and the baseline model is Figure 10b ​), in order to test the generalization ability of the framework, the present application shows the detection effect of the model on a brand new foggy image generated by the pix2pix model. The generation process of the test sample is as follows: first, train a pix2pix model using the paired data of Foggy Cityscapes, and then apply the trained model to a brand new sunny image that does not belong to the original data set. The results show that the model of the present application corrects some problems existing in the baseline model, such as solving the repeated detection of vehicle dense area, correcting the wrong detection of cyclists, and identifying traffic lights ignored by the baseline model. These improvements show that the framework of the present application helps to improve the perception ability of dynamic targets and traffic elements.

[0117] 2) Qualitative (visual) analysis: the present application can also perform visual analysis on the internal feature maps and weight maps of the model, as shown in Figures 11a-11f , thereby clearly revealing its internal rigorous and interpretable logic processing flow.

[0118] First, take the final detection result of Figure 11a as an example. Behind its superior performance is a complete and interlocking feature enhancement process of the present application, including the following three stages:

[0119] First stage: Degraded area positioning based on physical model

[0120] The basis of this method is that the model has the ability to identify and quantify the spatial area of information impairment caused by fog. Figure 11b The physical unreliability weight map generated by the inference-time fusion adapter according to the principle of atmospheric scattering model is shown. According to the atmospheric scattering model, the transmittance of each point on the pseudo-transmittance map t guide The transmittance of each point on the pseudo-transmittance map determines the degree of attenuation of scene radiation, so 1- t can be used as a physical measure of signal degradation. The higher weight value (highlighted area) in the figure represents the area where the model judges that the fog concentration is higher and the reliability of the original visual information is lower. It can be seen that this weight map locates the heavy fog area in the input image and depicts the transition of fog concentration from the distance to the near view. This weight map acts as a spatial modulator with physical meaning in the subsequent process.

[0121] Second stage: Effectiveness of privileged knowledge transfer

[0122] In order to evaluate the learning effect of the inference-time fusion adapter, the present application compares the pseudo-sunny feature intensity map ( Figure 11c ) output by it with the benchmark reference ( Figure 11dA comparison was made. Although the two methods differ in the absolute intensity of feature activations due to differences in network structure, they exhibit similarity in preserving high-level semantic information, such as the spatial distribution, contours, and structural details of objects. This structural similarity suggests that the inference-time fusion adapter, through a teacher-student learning paradigm, has learned the ability to infer potential sunny-day feature representations from degenerate inputs, thus providing an effective information source for the subsequent feature repair stage.

[0123] Phase 3: Evaluation of the effectiveness of spatial adaptive feature enhancement

[0124] Finally, this application compares the feature maps of the last (highest) layer output by the encoder before and after processing by the PI-CAF module. Figure 11e and Figure 11f The effectiveness of its spatial adaptive enhancement is evaluated by using [a specific method / mechanism]. The core observation is the region where the most significant change occurs between the feature maps before and after fusion, and the region corresponding to the physical weight map ([the specific region / mechanism]). Figure 11b The high unreliability (i.e., dense fog) areas marked by the physical weight map correspond to the high degree of unreliability. Specifically, the physical weight map ( Figure 11b The highlighted red area in the original foggy weather feature map () Figure 11e In the PI-CAF module, it appears as a specific activation mode. After fusion, in the fused feature map ( Figure 11f At the same spatial location, the pattern was significantly reshaped into high-intensity cyan and pink activation regions. In contrast, the regions marked as reliable in the physical weight map (blue areas) showed relatively mild changes in features before and after fusion. This phenomenon intuitively verifies the dual purpose of the core design of this application. First, it uses physical priors as guidance to concentrate feature enhancement resources on the regions with the most severe signal degradation. Second, it is also an effective risk control mechanism: considering that the pseudo-sunny features used during inference are inferences rather than real data and may be inaccurate, the module reduces its reliance on pseudo-sunny features in regions where the original information is clear and reliable, and trusts the original features more, thereby avoiding the introduction of potential artifacts or errors.

[0125] In summary, this series of visualized intermediate results intuitively demonstrates the workflow of "physical diagnosis → reference generation → targeted repair" in this application. This physically-perceptive, interpretable, and spatially adaptive enhancement mechanism is the theoretical and practical basis for the superior detection performance achieved in this application.

[0126] It can be seen that the application fundamentally solves the inconsistency problem between training and inference stages by introducing a unique inference-time fusion adapter module, and realizes high feasibility of engineering deployment. Through the imitation learning mechanism, the teacher-student learning paradigm originally dependent on privileged information (such as sunny day images) can run in an efficient single-stream architecture during inference, eliminating the limitations of traditional methods during deployment.

[0127] In terms of feature enhancement, the innovative PI-CAF module of the application deeply integrates the atmospheric scattering physical model with the cross-attention mechanism, and realizes spatially adaptive targeted repair through physically interpretable unreliability weight maps. This design not only breaks through the limitations of black box models, but also visually demonstrates the physical basis of feature repair, providing reliable technical support for safety-critical fields.

[0128] In terms of teaching paradigm, the application breaks through the answer-oriented mode of traditional knowledge distillation and adopts a high-level paradigm of imparting method. The teacher network no longer directly outputs results, but provides methodologies for generating ideal sunny day features and physical transmittance maps and other key intermediate components, and the student network realizes deep knowledge transfer by mastering the generation logic of these methodologies. This mode significantly improves the generalization ability of knowledge.

[0129] For the detection problem in foggy environment, the scheme specially optimizes the feature degradation problem of small and medium-sized objects. Experiments show that the detection accuracy of small and medium-sized objects is improved significantly, and the relative improvement of small object detection performance reaches the industry leading level. This breakthrough has decisive significance for the identification of key targets such as long-distance pedestrians and motorcycles in autonomous driving and other scenarios.

[0130] From the perspective of system safety, the scheme builds an all-weather stable environmental perception module to provide continuous and reliable target monitoring input for downstream control systems. This design of ensuring data stability from the perception source effectively avoids the systemic risks caused by environmental perception failure, and builds a solid technical foundation for safety-sensitive applications such as autonomous driving.

[0131] It should also be noted that the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0132] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A target detection method characterized by, The target detection method comprises: extracting first inference features from an input foggy image through a main detection stream; inputting the first inference features into an inference-time fusion adapter in the main detection stream to generate second inference features and a pseudo transmittance map; performing physical guidance-based cross-attention fusion on the first inference features and the second inference features through a physical information guidance cross-attention fusion module in the main detection stream by taking the pseudo transmittance map as a physical space guide to generate enhanced inference features and output a predicted target detection result, wherein the procedure of the physical guidance-based cross-attention fusion performed by the physical information guidance cross-attention fusion module comprises: Based on the transmittance at each point on the pseudo-transmittance map t guide Determine the physical unreliability weight for each point. W unreliability =1- t guide ; with the first inference feature F foggy performing a cross-attention operation with Query being the second inference feature and Key and Value being the first inference feature, to generate an attention-enhanced feature F attended ; and based on the physical unreliability weight W unreliability , the first inference feature F foggy with the attention enhancement feature F attended to determine an enhanced inference feature for the each point F fused = F foggy · (1- W unreliability )+ F attended · W unreliability .

2. The object detection method of claim 1, wherein, The main detection stream is determined through an imitation learning training process, comprising: obtaining a pair of training foggy image and training sunny image; processing the training foggy image through the main detection stream to extract first training features; processing the training sunny image through an auxiliary guidance stream to generate real features and a predicted transmittance map as teacher guidance signals; inputting the first training features into the inference-time fusion adapter to generate second training features and a pseudo transmittance map as student outputs; performing physical guidance-based cross-attention fusion on the first training features by using the teacher guidance signals to generate enhanced training features and output a predicted target training result; and constructing a composite loss function comprising a physical supervision loss, a feature consistency loss and an imitation loss, and performing end-to-end joint optimization on the main detection stream with the composite loss function minimized as an optimization objective, wherein the physical supervision loss is determined by a theoretical real transmittance map and the predicted transmittance map, the imitation loss is determined by the real features and the second training features and the predicted transmittance map and the pseudo transmittance map, the feature consistency loss is determined by the enhanced training features and the real features.

3. The object detection method of claim 2, wherein, The theoretical real transmittance map is determined by the following steps: determining a global atmospheric light value from the training foggy image through a dark channel prior algorithm; and according to an atmospheric scattering model, the training foggy image, the training sunny image and the global atmospheric light value, the theoretical real transmittance map is inversely solved through algebraic transformation.

4. The object detection method of claim 2, wherein, The feature consistency loss is determined by comparing the enhanced training features and the real features based on a pre-trained VGG network, and the difference between the activation maps output by one or more intermediate convolutional layers of the VGG network.

5. The object detection method of claim 2, wherein, The composite loss function further comprises a detection loss, the composite loss function L total The expression is: L total = L det + λ feat · L feat + λ phy · L phy + λ adapter · L adapter wherein, L det is the detection loss; L feat is the feature consistency loss; L phy is the physics supervision loss, determined based on an inverse atmospheric scattering model; L adapter is the imitation loss; λ feat , λ phy , λ adapter are weight coefficients corresponding to each loss, respectively.

6. A target detection system characterized by, The target detection system comprises: an auxiliary guidance stream for receiving a training sunny image in a training phase and generating real features and a predicted transmittance map as teacher guidance signals; and a main detection stream configured to, in the training phase, receive training foggy images paired with the training sunny images and extract first training features in the training foggy images, generate training enhanced features based on the teacher guidance signal guiding the first training features, and in the inference phase, receive an input foggy image to generate enhanced inference features and output a final target detection result, wherein the main detection stream further comprises: a feature extraction module configured to extract first inference features from the foggy image; an inference-time fusion adapter configured to generate second inference features and a pseudo-transmittance map based on the first inference features; a physical information guided cross-attention fusion module configured to use the pseudo-transmittance map as a physical space guide, perform physical guide-based cross-attention fusion on the first inference features and the second inference features, and generate enhanced inference features; and a decoding and prediction module configured to output a predicted target detection result based on the enhanced inference features, wherein the physical information guided cross-attention fusion module performs the process of the physical guide-based cross-attention fusion, which comprises: determining a physical unreliability weight for each point on the pseudo-transmittance map t guide determining a physical unreliability weight for each point on the pseudo-transmittance map W unreliability =1- t guide ; with the first reasoning feature as the Query, and the second reasoning feature as the Key and Value, performs a cross-attention operation to generate an attention-enhanced feature F foggy with the first reasoning feature as the Query, and the second reasoning feature as the Key and Value, performs a cross-attention operation to generate an attention-enhanced feature F attended ; and based on the physical unreliability weight W unreliability , the first inference feature F foggy with the attention enhancement feature F attended to determine an enhanced inference feature for the each point F fused = F foggy ·(1- W unreliability )+ F attended · W unreliability .

7. A visual environment monitoring device for a vehicle, characterized in that the monitoring device comprises: an image acquisition module configured to acquire a video stream in front of or around a vehicle as a raw input signal; a core processing unit configured to perform target detection on the raw input signal according to the target detection method of any one of claims 1-5 to generate monitoring information of the vehicle; and a data interface and output module configured to output a control instruction for the vehicle according to the monitoring information.

8. A vehicle characterized by comprising: comprises: the target detection system of claim 6; or the visual environment monitoring device of claim 7.

Images