A method and apparatus for detecting fog-penetrating targets based on multi-sensor fusion

Abstract

This invention discloses a method and apparatus for fog-penetrating target detection based on multi-sensor fusion. The method includes: configuring a sensor group; aligning the data collected by each sensor in the sensor group in terms of time and space; performing multimodal data representation on the aligned sensor data to obtain representation data; and using a pre-trained adaptive deep fusion model based on the representation data to locate the moving target and complete target detection. This invention enhances the system's perception capability by fusing effective multimodal and multi-scale information from various sensors, achieving end-to-end detection of unknown moving targets under various fog conditions. The introduction of an event camera enables accurate capture of unknown moving targets in challenging scenarios such as high-speed movement and sudden changes in illumination. The use of event camera data to generate an attention map for adaptive fusion further improves the model's detection performance.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a method and device for detecting fog-penetrating targets based on multi-sensor fusion. Background Technology

[0002] Currently, the development of maritime video surveillance systems is still in its early stages, mainly limited by factors such as the complex marine environment, severe weather (e.g., fog, thunderstorms, strong winds), and the numerous uncertainties associated with unknown moving targets. In clear weather, ordinary cameras and intelligent algorithms can effectively monitor and protect key areas. However, in severe weather, especially at sea with fog and high humidity, the detection effectiveness of ordinary cameras is significantly reduced. Furthermore, existing image training datasets are biased towards clear weather, and existing high-performance target detection networks are designed under the assumption of consistent and redundant data flow. This leads to the failure of existing technologies in foggy conditions due to data asymmetry (e.g., millimeter-wave radar can detect data in dense fog, but lidar detection is limited). Therefore, efficiently, accurately, and in real-time 24 / 7 locating and tracking unknown moving targets in complex and ever-changing maritime environments is a crucial foundation for achieving intelligent maritime monitoring.

[0003] To advance the development of target detection, existing technologies have proposed target detection methods for foggy conditions. For example, Chinese patent document CN 111161160 A discloses a fog obstacle detection method. The main steps are: acquiring the original foggy image of the road segment to be detected; inputting the original foggy image into a defogging model to perform defogging processing, obtaining an initial defogging image; wherein the defogging model is trained on a first convolutional neural network based on original fog-free sample images; post-processing the initial defogging image to obtain a first defogging image; inputting the first defogging image into an obstacle detection model to determine the position and outline of obstacles in the first defogging image; wherein the obstacle detection model is trained on a second convolutional neural network based on defogging sample images. However, this method cannot solve the target detection problem in dense fog, and can only handle target detection in light fog using only camera data. Furthermore, its perception capability is weak in challenging environments such as rapid movement, motion blur, and sudden changes in lighting, limiting its application scenarios. It also cannot solve the detection failure problem caused by data asymmetry in heavy fog. Summary of the Invention

[0004] One of the objectives of this invention is to provide a fog-penetrating target detection method based on multi-sensor fusion, thereby improving the performance of moving target capture in complex environments.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A fog-penetrating target detection method based on multi-sensor fusion, the method comprising:

[0007] A sensor group is configured, which includes a binocular camera, a far-infrared camera, an event camera, and a millimeter-wave radar. The sensor group has a two-layer arrangement structure, with the millimeter-wave radar mounted in the center of the upper layer and the event camera, binocular camera, and far-infrared camera mounted from left to right in the lower layer.

[0008] Align the data collected by each sensor in the sensor group in terms of time and space;

[0009] Multimodal data characterization is performed on the aligned data from each sensor to obtain characterization data;

[0010] Based on the representation data, a pre-trained adaptive deep fusion model is used to locate moving targets and complete target detection. The adaptive deep fusion model extracts features from the representation data corresponding to the binocular camera, far-infrared camera and millimeter-wave radar and stacks them to obtain a fused feature map. At the same time, an attention map based on the representation data corresponding to the event camera is constructed using an attention mechanism. The attention map is multiplied with the fused feature map to obtain an adaptive fusion feature for locating moving targets.

[0011] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0012] Preferably, the step of spatially aligning the data collected by each sensor in the sensor group includes: unifying the far-infrared camera, event camera, and millimeter-wave radar into the binocular camera coordinate system using geometric space transformation and calibration, and establishing a mapping relationship between them and the binocular camera; then mapping the measurement results of the far-infrared camera, event camera, and millimeter-wave radar onto each pixel of the binocular camera.

[0013] Preferably, the step of aligning the data collected by each sensor in the sensor group in terms of time includes: using the acquisition time of the sensor with the lowest acquisition frequency among the binocular camera, far-infrared camera, event camera and millimeter-wave radar as the standard, and acquiring the data of other sensors at the same time.

[0014] Preferably, the step of performing multimodal data characterization on the aligned sensor data to obtain characterization data includes:

[0015] For a stereo camera, the data structure representing the original color image output by the stereo camera is [W]. s Hs C s ], representing the width, height, and number of channels of the stereo camera image, respectively;

[0016] For far-infrared cameras, the thermal image output by the far-infrared camera is taken, and the resolution of the thermal image is enlarged to match the original color image of the binocular camera using image interpolation methods, with pixels as the unit. The data structure is [W f H f C f ], representing the width, height, and number of channels of the far-infrared camera image, respectively;

[0017] For millimeter-wave radar, the point cloud data output by the millimeter-wave radar is taken, and bilinear interpolation is performed on the point cloud data until it matches the original color image data of the stereo camera, that is, the tensor structure image-like data is formed, and the data structure is represented by [L,S,R], which represent the range, velocity and radar cross section values ​​respectively.

[0018] For event cameras, the asynchronous time series output by the event camera is taken, and an end-to-end data representation method is used to generate a voxel grid with time characteristics.

[0019] Preferably, the number of channels C in the color raw image representation data structure output by the binocular camera is... s =3, C in the thermal image characterization data structure output by the far-infrared camera f =1.

[0020] Preferably, the adaptive deep fusion model uses a ResNet34 neural network to extract features from the characterization data of millimeter-wave radar, binocular camera and far-infrared camera to obtain feature maps, and uses concatenation to stack multiple feature maps to obtain a fused feature map.

[0021] Preferably, multiplying the attention map with the fused feature map includes:

[0022] The corresponding pixel value in the fused feature map is weighted using each pixel value in the attention map.

[0023] Preferably, the adaptive deep fusion model uses a 1x1 convolution to reduce the dimensionality of the adaptive fusion feature channels to 256, and then uses two 3x3 convolutions in parallel, and adds an IOU branch to the regression branch to achieve classification and regression to obtain the moving target respectively.

[0024] Preferably, the loss function design of the adaptive deep fusion model includes:

[0025] The loss function is defined as the sum of the classification loss function and the regression loss function, that is:

[0026] L = L cls +αL reg

[0027] In the formula, L cls and L reg These are the classification loss function and the regression loss function, respectively, with α being the weight parameter;

[0028] Classification loss function L cls The binary cross-entropy between class confidence and label is defined as follows:

[0029]

[0030] In the formula, the loss is split into positive anchor boxes and negative anchor boxes, with a matching threshold of 0.5, p c To predict classification probabilities, l c Given true binary labels and N as the number of samples, Huber loss is used to regress the bounding box for each positive anchor box.

[0031]

[0032]

[0033] In the formula, N pos H(p) is the number of positive samples. c ,l c ) is the Huber loss function.

[0034] The fog-penetrating target detection method provided by this invention enhances the system's perception capability by fusing effective information from multiple sensors across multiple modalities and scales. It achieves end-to-end detection of unknown moving targets under various fog conditions, addressing the weakness of traditional cameras in low-visibility fog. The introduction of an event camera enables accurate capture of unknown moving targets in challenging scenarios such as high-speed movement and sudden changes in illumination, solving the problem of rapid detection of unknown moving targets in complex conditions like high-speed movement and changing illumination in foggy weather. Furthermore, the use of event camera data to generate attention maps enables adaptive fusion, resolving the problem of deep feature fusion from multiple sensors under data asymmetry and further improving the model's detection performance.

[0035] The second objective of this invention is to provide a fog-penetrating target detection device based on multi-sensor fusion, which improves the performance of capturing moving targets in complex environments.

[0036] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0037] A fog-penetrating target detection device based on multi-sensor fusion includes a processor and a memory storing a number of computer instructions, which, when executed by the processor, implement the steps of the fog-penetrating target detection method based on multi-sensor fusion. Attached Figure Description

[0038] Figure 1 The flowchart shows the fog-penetrating target detection method based on multi-sensor fusion of the present invention.

[0039] Figure 2 A schematic diagram showing the data characterization of the data from each sensor in this invention;

[0040] Figure 3 This is a schematic diagram of the adaptive deep fusion detection network of the present invention;

[0041] Figure 4 This is the target detection map output by the fog-penetrating target detection method based on multi-sensor fusion of the present invention in the experiment of the present invention for fog-penetrating target detection in dense fog environment;

[0042] Figure 5 This is the target detection map output by the fog-penetrating target detection method based on multi-sensor fusion of the present invention in another dense fog environment during the experiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0045] To address the low accuracy of moving target detection in existing technologies, particularly the inability to capture moving targets in foggy weather, this embodiment proposes a multi-sensor fusion approach employing traditional cameras, far-infrared cameras, event cameras, and millimeter-wave radar. Infrared imaging offers significant advantages such as strong smoke penetration, excellent nighttime surveillance performance, resistance to electromagnetic interference, and strong concealment and anti-interference capabilities. Millimeter-wave radar provides good detection of the azimuth and velocity of moving targets. Event cameras can track and image high-speed moving objects. This embodiment integrates the advantages of various sensors under different weather conditions, researching unknown moving target detection technology for situations with low visibility and blurred moving objects in foggy weather, aiming to achieve the detection and behavioral analysis of unknown moving targets in key areas or around floating islands.

[0046] like Figure 1 As shown, the fog-penetrating target detection method based on multi-sensor fusion in this embodiment includes:

[0047] Step 1: Configure the sensor group.

[0048] Multimodal perception is crucial for detecting unknown moving targets in foggy weather. Considering the inherent characteristics of each sensor, this embodiment employs a multi-sensor fusion array consisting of a traditional binocular camera, a far-infrared camera, an event camera, and a millimeter-wave radar. The overall structure is divided into two layers: the upper layer centrally houses the millimeter-wave radar, while the lower layer, from left to right, houses the event camera, binocular camera, and far-infrared camera, respectively, to enhance perception capabilities in severe weather conditions such as fog. Furthermore, efficient sealing is used to achieve resistance to salt spray and corrosion.

[0049] In one embodiment, to ensure the target detection effect, the following sensors are selected:

[0050] The binocular camera uses an RCCB camera, with both cameras having a resolution of 1920x1024, a baseline of 20.3cm, a focal length of 8mm, a field of view of 39.6x21.7 degrees, and a frequency of 30Hz.

[0051] The far-infrared camera uses an Axis Q1922 with a resolution of 640x480, a pixel pitch of 17um, an equivalent noise temperature difference of less than 100mK, and captures thermal images at a resolution of 30Hz.

[0052] The event camera uses Prophesee's Gen4, with a resolution of 1280x720, a time delay of 1µs (equivalent time accuracy >10000fps), a dynamic range of 120dB, and a power consumption of approximately 30mW.

[0053] The millimeter-wave radar uses a frequency-modulated continuous wave (FMCW) millimeter-wave radar with a frequency of 77 GHz, an angular resolution of 1°, a maximum range of 200 m, and performs position and velocity detection at a frequency of 15 Hz.

[0054] It should be noted that the above is only a preferred selection method provided in this embodiment. In other embodiments, the selection can be made according to actual needs.

[0055] Step 2: Align the data collected by each sensor in the sensor group in terms of time and space. This achieves data alignment and information association between the sensors in time and space, so as to fully utilize the information from each sensor at the later feature level.

[0056] Because different sensors are installed at different locations and have different sampling frequencies, there is a certain time and space difference between the data of each sensor. If this difference is not corrected, it will lead to data inconsistency. Therefore, time and space synchronization technology is used to align the data and associate the information of different sensors to ensure the sufficiency of feature-level fusion of each sensor in the later stage.

[0057] 1) Spatial data alignment: Using geometric spatial transformation and calibration, the far-infrared camera, event camera and millimeter-wave radar are unified into the binocular camera coordinate system and a mapping relationship with the binocular camera is established. Then, the measurement results of the far-infrared camera, event camera and millimeter-wave radar are mapped to each pixel of the binocular camera.

[0058] In this embodiment, when performing spatial data alignment, sensor data is uniformly mapped to the pixels of the stereo camera. The stereo camera has the following coordinate systems: world coordinate system (w), a reference coordinate system / datum coordinate system used to describe the position of the stereo camera and the object; and stereo camera coordinate system (c), fixed on the stereo camera, with its origin at the optical center, Z... c The axis is along the optical axis, X c and Y c The axes are parallel to the imaging plane; the image coordinate system (x, y) (referred to as the image coordinate system), expressed in physical units, has its origin at the intersection of the stereo camera's optical axis and the image plane, and its x and y axes are parallel to the stereo camera's X-axis. c Y c The axes are parallel and along the image plane; the image coordinate system (u,v) (abbreviated as pixel coordinate system) is expressed in pixels, with the origin at the upper left corner of the digital image, and the u and v axes are positively oriented to the right and downward along the image plane.

[0059] The coordinate mapping of each sensor is based on existing technology. This embodiment uses millimeter-wave radar and a stereo camera as examples for illustration. That is, the millimeter-wave radar coordinate system is transformed into a world coordinate system centered on the stereo camera, and then transformed into the pixel coordinate system through the stereo camera coordinate system and the image coordinate system respectively. The transformation relationship is as follows:

[0060]

[0061]

[0062] Among them, X r Y r Z r It is a millimeter-wave radar coordinate system, X w Y w Z w Using the world coordinate system, θ is the angle between the uv and x axes, X w0 ,Y w0 Let R and T represent the rotation and translation matrices, respectively. Let f represent the focal length of the binocular camera. Let uv be the pixel coordinate system with its origin at the top left corner of the image. Let u0 and v0 be the origins of the pixel coordinate system. Let dx and dy be the length and width of the pixel.

[0063] 2) Data alignment in time: The data from other sensors at the same time is obtained based on the acquisition time of the sensor with the lowest acquisition frequency among the binocular camera, far-infrared camera, event camera and millimeter-wave radar.

[0064] The inconsistency in data caused by different sensor sampling frequencies is particularly evident when the target is moving at high speed. In this embodiment, the frequency at which each sensor acquires data is collected, and the sensor with the lowest frequency is used as the time reference. That is, the time when the data is collected by the sensor is used as the time point to be inserted, and the data of other sensors at the same time is acquired. For example, according to the sensor parameters in step 1, the minimum sampling time of the millimeter-wave radar is 15Hz, so the time synchronization is based on the millimeter-wave radar.

[0065] Step 3: Perform multimodal data characterization on the aligned sensor data to obtain characterization data.

[0066] The prerequisite for deep multimodal fusion is data representation, that is, effective data encoding of data from each sensor (where the data structure after representation of each sensor data is as follows). Figure 2 As shown), the details are as follows:

[0067] For stereo cameras, the data structure representing the original color image (RGB image) output by the stereo camera is [W s H s C s ], representing the width, height, and number of channels of the stereo camera image, respectively. In this embodiment, C is taken as C. s =3.

[0068] For the far-infrared camera, the thermal image output by the far-infrared camera is used. Since the resolution of the far-infrared image is much lower than that of the binocular image, this embodiment uses image interpolation to enlarge the resolution of the thermal image to match the original color image of the binocular camera, using pixels as the unit. The data structure is [W f H f C f ], representing the width, height, and number of channels of the far-infrared camera image, respectively. In this embodiment, C is taken as C. f =1.

[0069] For millimeter-wave radar, data is typically presented as point clouds, including the target's range, velocity, azimuth, and RCS (radar cross section). Due to the sparse nature of radar data, its spatial and angular resolutions are low. Therefore, this embodiment takes the point cloud data output by the millimeter-wave radar and performs bilinear interpolation on the point cloud data until it matches the original color image data from the stereo camera, thus forming tensor-structured image-like data. The data structure is represented by [L, S, R], which represent the range, velocity, and radar cross section values, respectively.

[0070] For event cameras, they can effectively encode pixel-level brightness changes caused by moving objects in the scene, thereby generating asynchronous event sequences. This gives the sensor a very high dynamic range (>120dB) and high temporal resolution (in microseconds). These features effectively handle challenging scenarios such as rapid movement and sudden changes in illumination. When the target (or camera) moves, pixels asynchronously generate event sequences, forming a spatiotemporal point cloud to characterize the spatial distribution and dynamic behavior of the target. The i-th event point e i It can be defined as [x] i ,y i ,t i ,p i ], where x i y i and t i Represent the coordinates and timestamp of the event point, respectively, with polarity p. i ={-1,+1}, where -1 and +1 represent OFF and ON events, respectively. Since the event sequence is asynchronous and sparse, an end-to-end data representation method is adopted here. That is, a multilayer perceptron is used to encode the event time information of each event in the original event sequence, and the event values ​​that occur in the same spatial location are accumulated to form the final feature, thereby generating a voxel grid with time features.

[0071] Step 4: Based on the representation data, use a pre-trained adaptive deep fusion model to locate the moving target and complete the target detection.

[0072] This embodiment designs an end-to-end adaptive deep fusion detection network for the adaptive deep fusion model. It utilizes continuous convolutions to fuse feature information from different sensors across multiple modalities and scales. The feature extraction part of the backbone network employs a ResNet34 network, continuously increasing the number of channels in each layer through the layering of convolutional and pooling layers to reduce the feature map size. The feature map size decreases layer by layer, realizing a Feature Pyramid (FPN) for detection at different scales.

[0073] In this embodiment, the adaptive deep fusion model extracts features from the representation data corresponding to the binocular camera, far-infrared camera, and millimeter-wave radar and stacks them to obtain a fused feature map. At the same time, it uses an attention mechanism to construct an attention map based on the representation data corresponding to the event camera. The attention map is multiplied with the fused feature map to obtain an adaptive fusion feature for locating moving targets.

[0074] Adaptive deep fusion detection network structure as follows Figure 3 As shown, the entire network includes four data streams: event camera, millimeter-wave radar, binocular camera, and far-infrared camera. ResNet34 is used to extract features from the sensor data of millimeter-wave radar, binocular camera, and far-infrared camera to obtain feature maps. Multiple input feature maps are stacked using concatenation to achieve deep fusion of multi-size and multi-modal data to obtain fused feature maps.

[0075] Since event cameras have an inherent advantage in detecting moving targets in challenging environments such as high-speed movement and sudden changes in lighting, they are used as additional auxiliary data. An attention map based on the event camera is constructed using an attention mechanism. Since the pixel values ​​on the attention map reflect the amount of local information and can directly reflect the importance of the region, an attention fusion method is used to multiply it with the fusion feature map generated by the fusion of the other three sensors. That is, each pixel value on the attention map is used to weight the corresponding pixel in the fusion feature map, thereby achieving adaptive feature fusion to obtain adaptive fusion features.

[0076] In addition, this embodiment uses a relatively simple detection head to improve the convergence speed of the model. After trade-offs, a 1x1 convolution is used to reduce the dimensionality of the adaptive fusion feature channels to 256, and then two 3x3 convolutions are used in parallel. An IOU branch is added to the regression branch to achieve classification and regression respectively.

[0077] For the adaptive deep fusion model, the loss function designed in this embodiment is as follows:

[0078] The loss function is defined as the sum of the classification loss function and the regression loss function, that is:

[0079] L = L cls +αL reg

[0080] In the formula, L cls and L reg These are the classification loss function and the regression loss function, respectively, with α being the weight parameter;

[0081] Classification loss function L cls The binary cross-entropy between class confidence and label is defined as follows:

[0082]

[0083] In the formula, the loss is split into positive anchor boxes and negative anchor boxes, with a matching threshold of 0.5, p c To predict classification probabilities, l c Given true binary labels and N as the number of samples, Huber loss is used to regress the bounding box for each positive anchor box.

[0084]

[0085]

[0086] In the formula, N pos It is the number of positive samples, i.e., the number of positive anchor boxes, H(p) c ,l c ) is the Huber loss function. It's worth noting that in this embodiment, positive and negative samples are determined by their distance from the center of the ground truth target.

[0087] This embodiment pre-trains the proposed adaptive deep fusion detection network and uses a designed loss function for convergence during pre-training. The adaptive deep fusion model is obtained after pre-training. Network training is a standard procedure, and the training process will not be described in detail in this embodiment.

[0088] This embodiment addresses the challenge of target detection in dense fog by enhancing the system's perception capabilities using event cameras, far-infrared cameras, millimeter-wave radar, and binocular cameras. An adaptive deep fusion detection network is constructed to achieve multimodal and multi-scale deep feature fusion, thereby improving detection accuracy. To address the inability to detect rapidly moving targets, motion blur, and sudden changes in illumination in foggy conditions, an event camera is introduced, effectively utilizing its low latency and high dynamic range to resolve these issues. Furthermore, to address the detection failure caused by data asymmetry in foggy conditions, an event camera is used to achieve end-to-end local feature extraction via voxel networks and attention mechanisms, enabling adaptive deep feature fusion.

[0089] It is easy to understand that, based on the target detection method provided in this embodiment, a custom data grid can be used to characterize the data structure of the event camera, or other sensors such as LiDAR and gated cameras can be added to the overall system to achieve deformation, but all of the above deformations are within the protection scope of this application.

[0090] The effectiveness of the fog-penetrating target detection method of this application is further illustrated by the following experiment.

[0091] The experiment used DENSE as the training dataset, which includes 12,000 samples from over 10,000 kilometers of real-world driving scenarios in Northern Europe under various adverse weather conditions, and 1,500 samples with controllable indoor fog concentration. Figure 4 and 5 In the two sets of experimental data shown, the conventional target detection method YOLOv5 is unable to effectively detect the target (i.e., it cannot output the target bounding box as the original image, which is not shown). However, the fog-penetrating target detection method based on multi-sensor fusion in this application, due to the complementary advantages of multiple sensors, can achieve target detection under various foggy conditions. The output is as follows... Figure 4 and 5 The detection map shown has the target bounding box.

[0092] In another embodiment, this application also provides a fog-penetrating target detection device based on multi-sensor fusion, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the fog-penetrating target detection method based on multi-sensor fusion.

[0093] For specific limitations on fog-penetrating target detection devices based on multi-sensor fusion, please refer to the limitations on fog-penetrating target detection methods based on multi-sensor fusion mentioned above, which will not be repeated here.

[0094] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the fog-penetrating target detection method based on multi-sensor fusion in this embodiment of the invention by running the computer program stored in the memory.

[0095] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.

[0096] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] It should be noted that, Figure 4 , 5 The main purpose is to demonstrate the target detection effect of this embodiment in a dense fog environment. Figure 4 , 5 The graphics in the image are merely elements within the software's runtime interface and do not relate to the main focus of this application's improvements. Furthermore, the clarity of the runtime interface is related to pixels and scaling ratio, thus the presentation effect is relatively limited.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1.A method for fog-penetrating target detection based on multi-sensor fusion, characterized in that, The method comprises the following steps: The sensor group comprises a binocular camera, a far-infrared camera, an event camera, and a millimeter wave radar, and the sensor group is arranged in a two-layer structure, wherein the millimeter wave radar is arranged in the upper layer, and the event camera, the binocular camera, and the far-infrared camera are arranged in the lower layer from left to right; The data collected by each sensor in the sensor group is aligned in time and space; Each sensor data after data alignment is represented in a multi-modal data manner to obtain representation data; The representation data is used to locate a moving target by using a pre-trained adaptive deep fusion model to complete target detection, the adaptive deep fusion model extracts features from the representation data corresponding to the binocular camera, the far-infrared camera, and the millimeter wave radar, and stacks the features to obtain a fusion feature map, simultaneously, an attention map is constructed based on the representation data corresponding to the event camera by using an attention mechanism, the attention map is multiplied by the fusion feature map to obtain an adaptive fusion feature used for locating the moving target, and the adaptive deep fusion model uses a 1x1 convolution to reduce the channel of the adaptive fusion feature to 256, then uses two 3x3 convolutions in parallel, and adds an IOU branch to a regression branch to realize classification and regression respectively to obtain the moving target. 2.The method of claim 1, wherein, The data collected by each sensor in the sensor group is aligned in time and space; 3.The method of claim 1, wherein, The data collected by each sensor in the sensor group is aligned in time and space; 4.The method of claim 1, wherein, The data collected by each sensor in the sensor group is aligned in time and space; For binocular camera, take the color raw image representation data structure output by the binocular camera as [W s ,H s ,C s ], which respectively represent the width, height and channel number of the binocular camera image; For the far-infrared camera, the thermal image output by the far-infrared camera is taken, and the resolution of the thermal image is enlarged to be aligned with the color original image of the binocular camera by using an image interpolation method in units of pixels, and the data structure is represented as [W f ,H f ,C f ] representing the width, height and channel number of the far-infrared camera image respectively; The data collected by each sensor in the sensor group is aligned in time and space; For the millimeter wave radar, point cloud data output by the millimeter wave radar is taken, and bilinear interpolation is performed on the point cloud data until the point cloud data is matched with color original image data of the binocular camera, that is, image-like data in a tensor structure is formed, and the representation data structure is [L, S, R], which respectively represents distance, speed, and radar scattering cross-section value; 5.The method for detecting a see-through target based on multi-sensor fusion according to claim 4, wherein, The number of channels C in the color raw image representation data structure output by the binocular camera s = 3, and C f = 1 in the thermal image representation data structure output by the far infrared camera. 6.The method for detecting a see-through target based on multi-sensor fusion according to claim 1, wherein, For the event camera, an asynchronous time sequence output by the event camera is taken, and an end-to-end data representation method is used to generate a voxel grid with time features. 7.The method of claim 1, wherein, The adaptive deep fusion model uses a ResNet34 neural network to extract features from the representation data of the millimeter wave radar, the binocular camera, and the far-infrared camera to obtain a feature map, and uses concatenation to stack multiple feature maps to obtain a fusion feature map. The adaptive deep fusion model uses a ResNet34 neural network to extract features from the representation data of the millimeter wave radar, the binocular camera, and the far-infrared camera to obtain a feature map, and uses concatenation to stack multiple feature maps to obtain a fusion feature map. 8.The method for multi-sensor fusion based through fog target detection according to claim 1, wherein, The loss function of the adaptive deep fusion model comprises: The loss function is defined as the sum of a classification loss function and a regression loss function, that is, L = L cls + aL reg In the formula, L cls and L reg are a classification loss function and a regression loss function, respectively, and a is a weight parameter. The classification loss function L cls is the binary cross-entropy between class confidences and labels, defined as follows: where the loss is split into positive and negative anchor boxes, the matching threshold is 0.5, p c For predicting classification probabilities, l c where y is the true binary label, N is the number of samples, and for each positive anchor box, the Huber loss is used for bounding box regression: In the formula, N pos is the number of positive samples, H(p c , l c ) is the Huber loss function. 9.A device for detecting a see-through target based on multi-sensor fusion, comprising a processor and a memory storing a plurality of computer instructions, wherein, The computer instructions, when executed by a processor, implement the steps of the method for detecting a target through haze based on multi-sensor fusion according to any one of claims 1-8.

Images