Extreme environment target detection method based on cross-modal multi-scale fusion

Abstract

The invention provides an extreme environment target detection method based on cross-modal multi-scale fusion, which belongs to the technical field of target detection, and comprises the following steps: firstly, improving the data quality through data preprocessing; secondly, improving the position precision and semantic richness of virtual points through a multi-depth projection alignment strategy; a cascade interaction structure is constructed at different resolution voxel levels, and local details of high-resolution voxels and global context information of low-resolution voxels are fused through feature transmission. Under extreme conditions, the fragmented features of a distant target capture a local contour through low-level resolution voxels, and complement semantic features through high-level resolution voxels. And finally, target classification and detection are carried out under the view angle of the aerial view. Compared with a baseline method and other comparison methods, the method provided by the invention has obvious advantages in an extreme environment target detection task, and also has better detection performance in conventional weather.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to an extreme environment target detection method based on cross-modal multi-scale fusion. BACKGROUND

[0002] Single-modal target detection methods are affected by extreme environments, and have problems such as feature loss, data noise and data degradation, which are easy to cause missed detection and false detection. In contrast, multi-modal target detection methods effectively utilize three-dimensional coordinates, spatial forms, appearance textures and color features by fusing multi-modal data such as laser radar point clouds and camera images, thereby improving the data perception dimension.

[0003] Multi-modal fusion is a main method for solving the feature loss of single-modal data, and multi-modal data obtained by fusing sensors such as laser radars, millimeter wave radars and cameras are complementary to improve the environmental perception ability. However, in abnormal weather such as rain, snow and fog, the sensor data is distorted, and the accuracy of target detection decreases. In order to realize the reliability requirement of the detection task in extreme environments, the multi-modal data needs to be effectively filtered or repaired to remove the interference and noise influence of the extreme environment and improve the data quality.

[0004] In the three-dimensional target detection task, multi-modal fusion of laser radars and cameras provides complementary spatial perception and semantic information by combining the accurate geometric information of point clouds and the rich semantic features of images, thereby improving the perception ability in complex scenes. The bird's eye view has high efficient description ability for three-dimensional space, and is one of the mainstream frameworks for laser radar and camera fusion.

[0005] Deep learning-based object detection is generally divided into two-stage and one-stage methods. Two-stage methods generate candidate regions through methods such as region proposal networks, and then perform object positioning and classification within the candidate regions. Ross Girshick et al. proposed a region-based convolutional neural network that leverages the feature learning capabilities of R-CNN to improve the accuracy and performance of object detection, becoming a fundamental method in the field of computer vision. R-CNN gradually evolved into Fast R-CNN, Faster R-CNN, and other methods. One-stage detection methods only implement object classification and bounding box regression through a single stage, without the need to generate candidate regions. Tan et al. proposed Efficientdet, which uses a weighted bidirectional feature pyramid network and a composite scaling method to uniformly scale the resolution, depth, and width of all backbone networks, feature networks, and prediction networks. Duan et al. proposed CenterNet++, which detects each object as a three-tuple key point. By detecting angular key points, objects of different scales and shapes are detected, and center key points are used to further determine the target, reducing false positive confusion. The YOLO series of methods has made significant improvements in feature extraction, bounding box prediction, and other aspects with each version. Through a single forward propagation architecture, the object detection problem is transformed into an end-to-end fast regression problem, ensuring detection accuracy while improving real-time performance to meet the requirements of real-time video monitoring, autonomous driving auxiliary perception, and other applications that require speed.

[0006] While multimodal research in object detection has made significant progress, research in extreme environments is relatively limited. With the growing demand for high-precision and high-reliability object detection data, data processing under extreme conditions has become a challenging and practical research direction. In computer vision, image acquisition in extreme environments presents many challenges, such as reduced image contrast and blurred details caused by fog, and increased image noise and reduced object recognition caused by rain and snow. Physical model-based image dehazing is a major research approach. The dark channel prior algorithm proposed by He et al. effectively restores the structure of fog-free images by estimating atmospheric light values ​​and transmittance based on the statistical laws of dark channels in foggy environments, providing high-quality data for tasks such as object recognition and image analysis. CNN-based deep learning methods have become the leading approach for image dehazing. Ren et al. proposed a multi-scale convolutional neural network dehazing method that automatically learns feature representations for foggy images. Trained on large-scale foggy image data, it demonstrates strong dehazing capabilities. It performs well in scenes with uniform fog and can handle complex environments such as dense localized fog and dynamic fog changes, improving visual clarity. Research on image processing in rainy and snowy weather focuses on removing rain and snow streaks and enhancing target visibility. Traditional methods employ techniques such as morphological filtering and frequency domain transformation to detect and remove rain and snow noise texture from images, restoring the original image appearance. In recent years, deep learning-driven rain and snow removal methods have achieved remarkable results. Yang et al. constructed an end-to-end neural network architecture that inputs rainy and snowy images and outputs clear, rain-free images. This architecture surpasses traditional methods in terms of real-time performance and processing performance, effectively addressing the low availability of image data in extreme environments in fields such as security monitoring and autonomous driving. However, existing methods can improve point cloud and image quality to a certain extent, but they cannot guarantee the accuracy of target detection and are prone to over- or under-enhancement, resulting in unstable target detection performance. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this paper proposes a method for extreme environment target detection based on cross-modal multi-scale fusion. This method addresses the problem of poor data quality caused by point cloud noise in rainy and snowy weather environments through an outlier filtering method based on point cloud voxelization. Furthermore, an adaptive image enhancement method is introduced to enrich the color and texture information of the image, solving the problem of blurring the original image in foggy weather. A dual-branch network is used to extract features from the two modalities at different scales. Multi-depth projection alignment projects the point cloud into a two-dimensional image space to generate reference points, providing depth information for the image points and achieving data alignment. Multi-scale feature fusion uses a multi-scale feature interaction network to fuse features at each scale, completing the combination of multi-granularity information. Finally, the fused features are used for target detection through the BEV detection module. This paper was validated on four public datasets and found that the proposed model consistently outperformed baseline and single-modal target detection methods.

[0008] The present invention is implemented through the following technical solution: a method for detecting targets in extreme environments based on cross-modal multi-scale fusion: the method specifically comprises the following steps:

[0009] Step 1: Select a public dataset as the experimental dataset and divide it into training set and test set;

[0010] Step 2: De-noise the point cloud data using a voxel grid-based outlier filtering method;

[0011] Step 3: Repair the extreme environment image through parameter prediction network and adaptive image enhancement method;

[0012] Step 4: Extract geometric features and semantic features from the processed point cloud and image data respectively to provide semantic information for multimodal fusion object detection;

[0013] Step 5: Construct a multi-scale feature fusion model for LiDAR point cloud and camera image, perform multi-depth projection alignment through a three-stage projection strategy of reference point generation, depth estimation, and virtual point generation, and perform multi-modal fusion using multi-scale feature fusion technology.

[0014] Step 6: Use the test data divided in step 1 to perform target detection on the lidar point cloud and camera image multi-scale feature fusion model trained in step 5 to obtain the detection results.

[0015] Furthermore, in step 2,

[0016] Step 2.1: Convert the unordered and sparse point cloud data into a regular voxel grid, and approximate all points within each voxel with the centroid;

[0017] Step 2.2: Divide all centroid points into clusters through clustering, calculate the threshold by cluster, and retain the valid clusters;

[0018] Step 2.3: Output the filtered point cloud data.

[0019] Furthermore, in step 3,

[0020] Step 3.1: Downsample the image using bilinear interpolation to obtain a low-resolution image. Use a convolutional network to extract the global and local features of the image, and output the parameter prediction results through a fully connected network.

[0021] Step 3.2: Adaptively enhance the original input image by performing dehazing, gamma correction, contrast adjustment, and sharpening adjustment, and output the enhanced image.

[0022] Furthermore, in step 4,

[0023] Step 4.1: Voxelize the original point cloud to obtain a voxelized point cloud, perform feature encoding on the voxelized point cloud, extract point cloud features and splice them;

[0024] Step 4.2: Perform sparse convolution to obtain point cloud modal features, and perform multiple-stage feature extraction on the image through the feature extraction network, remove the features of the primary stage to avoid errors, and retain the features of the advanced stage.

[0025] Furthermore, in step 5,

[0026] Step 5.1: Convert the point cloud into a feature tensor, extract the 2D multi-level features of the image, and construct a candidate depth pool based on the reference points generated by projecting the lidar point cloud into 2D space;

[0027] Step 5.2: For each image point, retrieve its K nearest neighbor reference points on the image plane to obtain multiple candidate depth values. The weighted features and depths are projected into 3D space to generate virtual points and voxelized to a spatial resolution aligned with the lidar branch.

[0028] Step 5.3: Project the LiDAR point cloud onto the image plane to generate a set of candidate reference points. Based on the image point set in the plane space, use the K-nearest neighbor method to select K reference points around the candidate reference point set to generate a reference point set.

[0029] Step 5.4: Concatenate the camera image features with the depth of the reference points, and then generate depth-aware features through a convolutional layer. Voxelize the virtual points and the original lidar points into a unified spatial grid to obtain multi-scale voxel features.

[0030] Step 5.5: Through the gating mechanism and the progressive interaction between scales, the deep fusion of the lidar point cloud and the camera virtual point features is achieved to generate multimodal features with geometric and semantic information, and output features corresponding to different resolution scales.

[0031] Furthermore, in step 6,

[0032] Step 6.1: Compress the multi-scale fused voxel features along the height dimension to the BEV plane, and use channel weighted pooling to generate BEV features;

[0033] Step 6.2: Design a cross-level fusion mechanism to upsample BEV features of different scales to a unified resolution and then concatenate them;

[0034] Step 6.3: Use the BEV encoder to further extract the global context information in the BEV features and design a multi-task detection head based on the CenterPoint framework.

[0035] Furthermore, in step 6, the detection head outputs a classification branch and a regression branch;

[0036] The classification branch predicts the confidence of the possible location of the target, and the regression branch predicts the target attribute parameters and target direction.

[0037] A target detection system for extreme environments based on cross-modal multi-scale fusion:

[0038] The target detection system includes a preprocessing module, a multi-scale fusion module and a detection module:

[0039] The preprocessing module selects a public dataset as the experimental dataset and divides it into a training set and a test set; denoises the point cloud data using a voxel grid-based outlier filtering method; and repairs extreme environment images using a parameter prediction network and an adaptive image enhancement method.

[0040] The multi-scale fusion module extracts geometric features and semantic features from the pre-processed point cloud and image data respectively; constructs a multi-scale feature fusion model of lidar point cloud and camera image, performs multi-depth projection alignment through a three-stage projection strategy of reference point generation-depth estimation-virtual point generation, and uses multi-scale feature fusion technology to perform multimodal fusion;

[0041] The detection module uses the test data divided by the preprocessing module to perform target detection on the lidar point cloud and camera image multi-scale feature fusion model trained by the multi-scale fusion module to obtain a detection result.

[0042] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0043] A computer-readable storage medium is used to store computer instructions, which implement the steps of the above method when executed by a processor.

[0044] Beneficial effects of the present invention

[0045] Compared with the prior art, the present invention has at least the following advantages:

[0046] First, a filtering method based on Voxel grid and outlier removal was designed to effectively suppress the interference of rain and fog noise on point cloud data; in terms of image degradation enhancement, a parameter prediction network and adaptive image restoration were proposed, which significantly improved the clarity and contrast of images under harsh conditions such as low light and haze, providing a good data foundation for multimodal target detection in extreme environments.

[0047] Secondly, multi-depth projection alignment significantly improves the spatial alignment accuracy of image and point cloud data by guiding virtual point generation through multi-depth assumptions, demonstrating enhanced robustness in scenarios with extreme point cloud sparsity and depth uncertainty. Multi-scale feature fusion effectively alleviates the feature fragmentation problem of distant targets by complementary fusion of high-resolution geometric features and low-resolution semantic features, enhancing the overall perception of complex scenes. The introduction of a gating network enables adaptive weight allocation of features between modalities, effectively suppressing noise interference from low-quality modalities in extreme environments and improving the stability of fused features.

[0048] Experimental results show that this method has a significant advantage over baseline methods and other comparison methods in extreme environment target detection tasks, and also has better detection performance in normal weather conditions, which proves the generalization and superiority of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the extreme environment target detection method based on cross-modal multi-scale fusion of the present invention;

[0050] Figure 2 This is a diagram of the point cloud filtering method of the present invention;

[0051] Figure 3 Schematic diagram of the image adaptive enhancement method of the present invention;

[0052] Figure 4 This is a diagram of the extreme environment target detection method based on cross-modal multi-scale fusion of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] The experimental methods used in the following examples are conventional methods unless otherwise specified. The materials, reagents, methods, and instruments used are conventional in the art and can be obtained commercially by those skilled in the art unless otherwise specified.

[0055] like Figure 1As shown in the figure, the present invention's method for extreme environment target detection based on cross-modal multi-scale fusion removes noise points from the original point cloud through filtering based on voxel segmentation and outlier removal, providing noise-free point cloud data for multimodal target detection tasks. Adaptive image restoration is performed using parameters transmitted by a parameter prediction network, dehazing and adaptively correcting images in foggy and other low-light conditions, providing clear image data for multimodal target detection tasks. Furthermore, effective fusion of 3D point cloud data and 2D image data is achieved.

[0056] The specific steps include:

[0057] Step 1: Select public datasets (such as WADS, RTTS, VOC, and nuScenes) as experimental datasets and divide them into training and test sets.

[0058] Step 1.1: Use the WADS dataset for point cloud data experiments, use RTTS and VOC for image data experiments, and use the nuScenes dataset as the experimental dataset for the fusion model;

[0059] Step 1.2: Divide different data sets into training sets and test sets according to requirements;

[0060] Step 2: Figure 2 As shown in the figure, an outlier filtering method based on point cloud voxelization is used. Through voxelization processing and outlier removal, the divide-and-conquer idea is used to remove outliers through six steps: initialization, local calculation, global calculation, point cloud screening, cluster screening, and retaining valid points. This reduces data redundancy, retains the contour information of the target, and improves the quality and reliability of point cloud data.

[0061] Step 2.1: Point cloud voxelization: Divide the 3D point cloud space into voxel grids, that is, convert the disordered and sparse point cloud data into a regular grid structure. Each point p i =(x i ,y i ,z i ) is assigned to the corresponding voxel V u,v,w In , the index of each voxel is (u,v,w), and the calculation formula is as follows:

[0062]

[0063] Step 2.2: After voxelization, each voxel V u,v,w There may be multiple point cloud data points. To reduce data redundancy and retain the geometric information of the target, each V u,v,w All points in are approximated by a centroid:

[0064]

[0065] Step 2.3: Calculate the centroid n of all voxels j Combine to generate the down-sampled point cloud set P down :

[0066]

[0067] N down Indicates the number of down-sampled point cloud data.

[0068] Step 2.4: Cluster the point cloud space and assign the point cloud to the cluster;

[0069] Step 2.5: Perform local calculation, for each point p i , calculate the average squared Euclidean distance to the k nearest neighbors, the calculation formula is as follows:

[0070]

[0071] Step 2.6: Perform global calculation, calculate the average distance μ and standard deviation ∩ of all points in the global;

[0072] Step 2.7: Filter the point cloud, set the threshold α as the deletion of abnormal point division interval, to retain normal point cloud while reducing the complexity of calculation, further filter the retained cluster.

[0073]

[0074] Step 2.8: Keep the effective point P' = {p'1, p'2,... p' N'};

[0075] Step 3: As Figure 3 shown, based on YOLO network, through parameter prediction network and adaptive image enhancement method, realize the repair processing of extreme environment image, highlight the target features;

[0076] Extract the global features of the input image, and predict the required transmittance ω, contrast enhancement coefficient α and other parameters, and perform adaptive enhancement on the image based on the physical model;

[0077] Step 3.1: Down-sample by bilinear interpolation to get low-resolution image, the calculation formula is as follows:

[0078] I down = BilinearResize(I, 256x256)

[0079] Step 3.2: Use convolution network to extract global features and local features of the image, and output parameter prediction results through fully connected network:

[0080] fFC1 =LeakyReLU(W FC1 ·f flat +b FC1 )

[0081] P N =W FC2 ·f FC1 +b FC2

[0082] P N is an output parameter;

[0083] Step 3.3: Adaptively enhance the original input image according to the predicted parameters, and perform dehazing, gamma correction, contrast enhancement, and sharpening enhancement processing respectively. The calculation formula is as follows:

[0084]

[0085] I G (x) = c·I defog (x) γ

[0086] I contrast (x) = α·I G +β

[0087] I sharp =I contrast +λ·(I contrast -I blur )

[0088] I defog Represents the image after defogging, I G represents the corrected image, I contrast Represents the image after contrast adjustment, I sharp Represents the image after sharpening enhancement;

[0089] Step 3.4: Output the enhanced image.

[0090] Step 4: Extract high-quality geometric and semantic features from the preprocessed point cloud and image data. Point cloud feature extraction converts the three-dimensional spatial structure information collected by the LiDAR into quantitative features with clear spatial relationships. The point cloud is voxelized to generate regular grid cells and the geometric distribution of local key points is extracted. Image feature extraction captures visual details such as color, texture, and object outlines in the image, providing rich semantic information for multimodal fusion target detection.

[0091] Step 4.1: Voxelize the original point cloud P to obtain a voxelized point cloud with a size of (V D ,V H ,V W );

[0092] Step 4.2: Feature encoding of the voxelized point cloud. Extract point cloud features Concatenate features to get

[0093] Step 4.3: Sparse convolution to obtain point cloud modality features

[0094] Step 4.4: 5-stage feature extraction of the image by the feature extraction network, respectively and

[0095] Step 4.5: Remove the features of the primary stage to avoid errors and retain the features of the advanced stage

[0096] Step 5: Construct a multi-scale feature fusion model based on the laser radar point cloud and camera image. Through the three-stage projection strategy of reference point generation-depth estimation-virtual point generation, multi-depth projection alignment is performed, and multi-modal fusion is performed using multi-scale feature fusion technology. A cascaded interaction structure is constructed at different resolution voxel levels. The local details of high-resolution voxels and the global context information of low-resolution voxels are fused through feature transmission. The fragmented features of distant targets are captured by low-resolution voxels to complete the local contour, and the semantic features are completed by high-resolution voxels. The semantic correlation between three-dimensional space and two-dimensional image is fully explored.

[0097] Step 5.1: Convert the point cloud to a feature tensor F LiDAR ∈R V×C , where V is the number of encoded voxels, and C is the number of feature channels. Extract 2D multi-level features F image ∈R H'×W'×C , where H' and W' are the sizes of the downsampled feature maps.

[0098] Step 5.2: Uniformly sample image points from the pre-trained 2D instance mask, and combine the reference points projected to the 2D space by the laser radar point cloud to construct a candidate depth pool;

[0099] Step 5.3: For each image point, retrieve its K nearest neighbor reference points on the image plane to obtain multiple candidate depth values to cover the possible position distribution of the target object in 3D space.

[0100] Step 5.4: Weighted features and depth are projected to 3D space to generate virtual points, and voxelized to the spatial resolution aligned with the laser branch.

[0101] Step 5.5: P ∈ R N×4Represents the lidar point cloud, which is projected onto the image plane to generate a set of candidate reference points.

[0102] R cand ={(u i ,v i ,d i )}

[0103]

[0104] Step 5.6: Image point set in plane space The K-nearest neighbor method is used to select K reference points around the candidate reference point set, generate a reference point set, and obtain the depth set The reference point set can be expressed as:

[0105]

[0106] Step 5.7: Convert the camera image features F image It is concatenated with the depth of the reference point and then passes through the convolutional layer to generate the depth-aware feature F depth-aware ;

[0107] Step 5.8: Voxelize the virtual points and the original lidar points into a unified spatial grid to obtain multi-scale voxel features

[0108] Step 5.9: Through the gating mechanism and the gradual interaction between scales, the deep fusion of the lidar point cloud and the camera virtual point features is achieved to generate multimodal features with geometric and semantic information;

[0109] Step 5.10: Output contains L features corresponding to different resolution scales

[0110] Step 6: BEV detection solves the sensor's viewing angle limitation and occlusion problems by projecting objects in 3D space onto a 2D plane. Through high compression and multi-level feature fusion strategies, the multimodal voxel features output by the multi-scale feature fusion module are projected into the bird's-eye view space. The detection head then performs target positioning and attribute prediction to generate the final 3D detection results.

[0111] Step 6.1: Voxel features after multi-scale fusion Compress along the height dimension (Z axis) to the BEV plane, and use channel weighted pooling to generate BEV features;

[0112] Step 6.2: Design a cross-level fusion mechanism: Combine BEV features of different scales Upsample to a uniform resolution and then stitch;

[0113] Step 6.3: Use the BEV encoder to further extract global context information from the BEV features and design a multi-task detection head based on the CenterPoint framework, including classification branch and regression branch:

[0114] P cls =σ(Conv 1×1 (F BEV-fuse ))

[0115] P reg =Conv 1×1 (F BEV-fuse )

[0116] P dir =L2Norm(Conv 1×1 (F BEV-fuse ))

[0117] The confidence P of the classification branch predicting the possible location of the target cls , the regression branch predicts the target attribute parameter P reg and target direction;

[0118] A target detection system for extreme environments based on cross-modal multi-scale fusion:

[0119] The target detection system includes a preprocessing module, a multi-scale fusion module and a detection module:

[0120] The preprocessing module selects a public dataset as the experimental dataset and divides it into a training set and a test set; denoises the point cloud data using a voxel grid-based outlier filtering method; and repairs extreme environment images using a parameter prediction network and an adaptive image enhancement method.

[0121] The multi-scale fusion module extracts geometric features and semantic features from the pre-processed point cloud and image data respectively; constructs a multi-scale feature fusion model of lidar point cloud and camera image, performs multi-depth projection alignment through a three-stage projection strategy of reference point generation-depth estimation-virtual point generation, and uses multi-scale feature fusion technology to perform multimodal fusion;

[0122] The detection module uses the test data divided by the preprocessing module to perform target detection on the lidar point cloud and camera image multi-scale feature fusion model trained by the multi-scale fusion module to obtain a detection result.

[0123] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0124] A computer-readable storage medium is used to store computer instructions, which implement the steps of the above method when executed by a processor.

[0125] The memory in the embodiments of the present application can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to comprise, but not be limited to, these and any other suitable types of memory.

[0126] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired connection such as a coaxial cable, optical fiber, digital subscriber line (DSL), or wireless connection such as infrared, wireless, or microwave. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium such as a floppy disk, hard disk, magnetic tape, an optical medium such as a high-density digital video disc (DVD), or a semiconductor medium such as a solid-state disc (SSD).

[0127] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0128] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-described method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-described processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-described method.

[0129] The above is a detailed introduction to the extreme environment target detection method based on cross-modal multi-scale fusion proposed in the present invention, and the principles and implementation methods of the present invention are explained. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting targets in extreme environments based on cross-modal multi-scale fusion, characterized by: The method specifically comprises the following steps: Step 1: Select a public dataset as the experimental dataset and divide it into training set and test set; Step 2: De-noise the point cloud data using a voxel grid-based outlier filtering method; Step 3: Repair the extreme environment image through parameter prediction network and adaptive image enhancement method; Step 4: Extract geometric features and semantic features from the processed point cloud and image data respectively to provide semantic information for multimodal fusion object detection; Step 5: Construct a multi-scale feature fusion model for LiDAR point cloud and camera image, perform multi-depth projection alignment through a three-stage projection strategy of reference point generation, depth estimation, and virtual point generation, and use multi-scale feature fusion technology for multi-modal fusion. Step 6: Use the test data divided in step 1 to perform target detection on the lidar point cloud and camera image multi-scale feature fusion model trained in step 5 to obtain the detection results.

2. The detection method according to claim 1, wherein: In step 2, Step 2.1: Convert the unordered and sparse point cloud data into a regular voxel grid, and approximate all points within each voxel with the centroid; Step 2.2: Divide all centroid points into clusters through clustering, calculate the threshold by cluster, and retain the valid clusters; Step 2.3: Output the filtered point cloud data.

3. The detection method according to claim 2, characterized in that: In step 3, Step 3.1: Downsample the image using bilinear interpolation to obtain a low-resolution image. Use a convolutional network to extract the global and local features of the image, and output the parameter prediction results through a fully connected network. Step 3.2: Adaptively enhance the original input image by performing dehazing, gamma correction, contrast adjustment, and sharpening adjustment, and output the enhanced image.

4. The detection method according to claim 3, wherein: In step 4, Step 4.1: Voxelize the original point cloud to obtain a voxelized point cloud, perform feature encoding on the voxelized point cloud, extract point cloud features and splice them; Step 4.2: Perform sparse convolution to obtain point cloud modal features, and perform multiple-stage feature extraction on the image through the feature extraction network, remove the features of the primary stage to avoid errors, and retain the features of the advanced stage.

5. The detection method according to claim 4, characterized in that: In step 5, Step 5.1: Convert the point cloud into a feature tensor, extract the 2D multi-level features of the image, and construct a candidate depth pool based on the reference points generated by projecting the lidar point cloud into 2D space; Step 5.2: For each image point, retrieve its K nearest neighbor reference points on the image plane to obtain multiple candidate depth values. The weighted features and depths are projected into 3D space to generate virtual points and voxelized to a spatial resolution aligned with the lidar branch. Step 5.3: Project the LiDAR point cloud onto the image plane to generate a set of candidate reference points. Based on the image point set in the plane space, use the K-nearest neighbor method to select K reference points around the candidate reference point set to generate a reference point set. Step 5.4: Concatenate the camera image features with the depth of the reference points, and then generate depth-aware features through a convolutional layer. Voxelize the virtual points and the original lidar points into a unified spatial grid to obtain multi-scale voxel features. Step 5.5: Through the gating mechanism and the progressive interaction between scales, the deep fusion of the lidar point cloud and the camera virtual point features is achieved to generate multimodal features with geometric and semantic information, and output features corresponding to different resolution scales.

6. The detection method according to claim 5, characterized in that: In step 6, Step 6.1: Compress the multi-scale fused voxel features along the height dimension to the BEV plane, and use channel weighted pooling to generate BEV features; Step 6.2: Design a cross-level fusion mechanism to upsample BEV features of different scales to a unified resolution and then concatenate them; Step 6.3: Use the BEV encoder to further extract the global context information in the BEV features and design a multi-task detection head based on the CenterPoint framework.

7. The detection method according to claim 6, characterized in that: In step 6, the detection head outputs the classification branch and the regression branch; The classification branch predicts the confidence of the possible location of the target, and the regression branch predicts the target attribute parameters and target direction.

8. A target detection system for extreme environments based on cross-modal multi-scale fusion, characterized by: The system is used to execute the extreme environment target detection method according to any one of claims 1 to 7: The target detection system includes a preprocessing module, a multi-scale fusion module and a detection module: The preprocessing module selects a public dataset as the experimental dataset and divides it into a training set and a test set; denoises the point cloud data using a voxel grid-based outlier filtering method; and repairs extreme environment images using a parameter prediction network and an adaptive image enhancement method. The multi-scale fusion module extracts geometric features and semantic features from the pre-processed point cloud and image data respectively; constructs a multi-scale feature fusion model of lidar point cloud and camera image, performs multi-depth projection alignment through a three-stage projection strategy of reference point generation-depth estimation-virtual point generation, and uses multi-scale feature fusion technology to perform multimodal fusion; The detection module uses the test data divided by the preprocessing module to perform target detection on the lidar point cloud and camera image multi-scale feature fusion model trained by the multi-scale fusion module to obtain a detection result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images