Multi-modal 3D target detection method based on chiral perception fusion

Abstract

The invention provides a multi-modal 3D target detection method based on chiral perception fusion. The method comprises the following steps: converting pixel coordinates and prediction depth in obtained initial image data of a to-be-detected target into virtual points of a three-dimensional space through a virtual point generation network; randomly dividing the virtual points into a plurality of virtual point sets, and performing density chiral sampling on the virtual points in the virtual point sets; performing fusion processing on the virtual point after density chiral sampling and the initial point cloud data; carrying out voxelization processing on the fused point cloud data to obtain voxel units; and performing position chiral suppression processing on the voxel units, and detecting data after the position chiral suppression processing to obtain a detection result. According to the method, a chiral perception fusion mechanism is designed by using'density chirality 'and'position chirality', so that the model can more effectively process the problems of non-uniform distribution of virtual points and position deviation, the detection precision is improved, and a firmer technical support is provided for application scenes such as automatic driving and vehicle-road cooperation.

Description

Technical Field

[0001] The present invention relates to a multimodal 3D target detection method and device based on chirality perception fusion, belonging to the technical field of intersection of computer vision and biochemistry. Background Art

[0002] 3D object detection plays a key role in areas such as autonomous driving, vehicle-road collaboration, surface navigation, and human-machine collaboration. This technology relies primarily on two types of sensors: lidar and cameras. LiDAR uses point cloud technology to scan the surrounding environment, providing accurate distance information and target location. However, due to the distribution characteristics of its point cloud, where the point cloud is dense in close-range areas and relatively sparse in distant areas, the performance of lidar is reduced when detecting distant targets. On the other hand, although cameras can capture high-resolution images, providing rich semantic information and appearance details for scene analysis, they have limitations in accurately locating 3D objects.

[0003] Given these shortcomings of single-type sensors, multimodal fusion methods that combine the strengths of both lidar and cameras have begun to attract attention. Previous studies have attempted to combine semantic features from cameras with lidar point cloud data, improving object detection to some extent. However, these approaches often fail to fully account for the asymmetry between the two modalities, particularly addressing the sparseness of lidar point clouds at long distances. Recent research has proposed novel strategies, such as generating virtual or pseudo points from images to fill gaps in distant lidar point clouds, which significantly improves detection accuracy. However, the spatial distribution of virtual points generated from images exhibits unevenness. For example, in the KITTI dataset, this unevenness manifests as dense virtual points in close-range areas and sparse virtual points in distant areas. To address this uneven distribution, previous studies have adopted different strategies based on the distance between the object and the sensor. However, these methods often use strict binning to distinguish objects at different distance ranges. As a result, objects at bin boundaries may be affected by multiple strategies, resulting in discontinuities or breaks in features. Furthermore, the spatial distribution of 3D points can be erroneous, leading to significant positional deviations on the object surface. For example, objects such as vehicles and pedestrians often have inherently symmetrical structures. However, when combining LiDAR point clouds and image virtual points, these symmetries can be disrupted and improperly processed due to noise or occlusion during depth completion, reducing the accuracy of 3D object detection. Some early work attempted to address positional bias by using prior knowledge, but if the prior knowledge is not accurate or complete, it can lead to biased detection results or even missed detections. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal 3D target detection method and device based on chirality perception fusion, which realizes dynamic adaptive sampling by calculating density chirality sampling, so that the spatial distribution of sampled virtual points is more uniform; and then extracts chirality features through position chirality suppression processing to improve the accuracy of 3D target detection.

[0005] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] In one aspect, the present invention provides a multimodal 3D object detection method based on chirality perception fusion, comprising:

[0007] Obtain the initial point cloud data and image data of the target to be measured;

[0008] Input the point cloud data and image data into the trained target detection model for detection processing to obtain the detection results;

[0009] Wherein: the detection process in the trained target detection model specifically includes:

[0010] The two-dimensional coordinates of pixels in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network;

[0011] Randomly dividing the virtual points into a plurality of virtual point sets, and performing density chirality sampling on the virtual points in the virtual point sets;

[0012] Fuse the virtual points after density chirality sampling with the initial point cloud data;

[0013] The fused point cloud data is voxelized to obtain voxel units;

[0014] The voxel unit is subjected to position chirality suppression processing, and the data after the position chirality suppression processing is detected to obtain a detection result.

[0015] Furthermore, the two-dimensional coordinates of pixels in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network, specifically including:

[0016] Use the monocular depth estimation network to predict the image depth and obtain the predicted depth corresponding to the two-dimensional coordinates of each pixel in the initial image data;

[0017] According to the predicted depth, the coordinate transformation is performed by obtaining the camera intrinsic parameters and extrinsic parameters of the image data to convert the two-dimensional coordinates of the pixel into the three-dimensional space coordinates of the virtual point.

[0018] Furthermore, the density chirality sampling of the virtual points in the virtual point set specifically includes:

[0019] Calculate the density chirality characteristics of the virtual point, the expression is:

[0020] ;

[0021] ;

[0022] in, is a virtual point set containing virtual point i and its neighborhood virtual point j, is the convolution kernel weight at the neighborhood point j, is the gradient amplitude of the neighborhood virtual point j, is the density chirality characteristic at the virtual point i; Density chirality Normalized density numerical expression; is the density chirality characteristic at the virtual point t, represents the natural exponential function, N is the number of virtual point sets;

[0023] According to the density value of virtual point i Sort the virtual point set in descending order, and put the descending order before The first virtual point set is a dense area, and the remaining virtual point sets are sparse areas;

[0024] Step 3: Sample the virtual point set and retain the virtual point set in the dense area The M virtual points with the highest internal density values ​​retain the virtual point set in the sparse area All virtual points in the, expression is:

[0025] ;

[0026] ;

[0027] in, represents the concatenation operation of a set of virtual points, The sum of the virtual point sets reserved for dense areas, is the sum of the virtual point sets reserved for the sparse area, and M is the virtual point set in the dense area The number of virtual points retained in is the i-th virtual point set in the dense area, is the i-th virtual point set in the sparse area, is the separation threshold.

[0028] Furthermore, the separation threshold The calculation method is:

[0029] ;

[0030] in, is the density value corresponding to the virtual point i, The function is used to find the maximum k value that satisfies the inequality, where k is a hyperparameter and τ is the chiral density accumulation threshold.

[0031] Furthermore, the fusion processing of the virtual points after density chirality sampling with the initial point cloud data specifically includes:

[0032] ;

[0033] F is the fused point cloud data, The sum of the virtual point sets reserved for dense areas, The sum of the virtual point sets retained for the sparse area, L is the point in the initial point cloud data.

[0034] Furthermore, the fused point cloud data is voxelized to obtain voxel units, which specifically includes:

[0035] The fused point cloud data is divided into voxel units of fixed size, and the expression is:

[0036] ;

[0037] Where: T is the voxel unit, It is a voxelization operation.

[0038] Furthermore, the step of performing position chirality suppression processing on the voxel unit and performing detection based on the processed data specifically includes:

[0039] For each voxel unit T, define its local neighborhood, calculate the symmetry difference, and extract the position chirality feature , the expression is:

[0040] ;

[0041] in, is the chirality feature of the mth voxel unit position, is the neighboring voxel of the mth voxel unit Entropy of the distribution; is the balance coefficient, which is used to control the weight of entropy. is the neighboring voxel of the mth voxel unit The coordinates of Indicates the calculation of neighborhood voxel coordinates Its symmetrical voxel coordinates relative distance;

[0042] Among them, the voxel coordinates The specific calculation method is: learn the symmetry plane parameters through a lightweight multi-layer perceptron and perform adaptive symmetry plane transformation. The expression is:

[0043] ;

[0044] in, is the rotation matrix; is the translation vector;

[0045] The 3D geometric features and 2D image features in the initial point cloud data and image data are obtained through the virtual convolutional network respectively;

[0046] The 3D geometric features, 2D image features, and chirality features are dynamically weighted and fused through the attention mechanism, and the dynamic weighted fusion features are detected. The expression is:

[0047] ;

[0048] ;

[0049] in: is the dynamic weighted fusion feature, is the 3D geometric feature of the voxel unit containing the virtual point n, is the 2D image feature corresponding to the 3D geometric feature, Represents a learnable weight matrix with dimensions , represents the adaptive weight coefficient, is the Sigmoid function, is a learnable parameter, Represents feature concatenation operation;

[0050] The obtained dynamic weighted fusion features are used to generate proposals through the region proposal network, extract the features of the region of interest, and then processed by the detection head to finally obtain the target detection results.

[0051] Furthermore, the training process of the target detection model specifically includes:

[0052] Obtain historical point cloud data, image data and corresponding target annotation information;

[0053] Construct a training set based on historical point cloud data, image data, and corresponding target annotation information, and divide the training data into several batches;

[0054] Input the training set into the target detection model in batches for training;

[0055] During the training process, the parameters of the target detection model are adjusted through the optimization algorithm to minimize the difference between the predicted results and the actual target annotation information;

[0056] After multiple rounds of iterative training, until the target detection model converges, a trained target detection model is obtained.

[0057] In a second aspect, the present invention provides a multimodal 3D object detection device based on chirality perception fusion, comprising:

[0058] An acquisition module is used to obtain the initial point cloud data and image data of the target to be measured;

[0059] The processing module is used to input the point cloud data and image data into the trained target detection model for detection processing and obtain the detection results;

[0060] Wherein: the detection process in the trained target detection model specifically includes:

[0061] The two-dimensional coordinates of pixels in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network;

[0062] Randomly dividing the virtual points into a plurality of virtual point sets, and performing density chirality sampling on the virtual points in the virtual point sets;

[0063] Fuse the virtual points after density chirality sampling with the initial point cloud data;

[0064] The fused point cloud data is voxelized to obtain voxel units;

[0065] The voxel unit is subjected to position chirality suppression processing, and the data after the position chirality suppression processing is detected.

[0066] In a third aspect, the present invention provides a multimodal 3D object detection system based on chirality perception fusion, comprising:

[0067] Memory, used to store computer programs / instructions;

[0068] A processor is used to execute the computer program / instructions to implement the steps of the multimodal 3D target detection method based on chirality perception fusion.

[0069] Compared with the existing technology, the beneficial effects achieved by the present invention are: from the perspective of suppressing noise interference and improving detection accuracy, the present invention introduces "density chirality" and "position chirality" of virtual points, so that the model can more effectively deal with the problems of uneven distribution and position deviation of virtual points; in complex scenes, such as in the presence of occlusion and noise interference, compared with traditional methods, the chirality perception fusion mechanism adopted by the present invention can effectively reduce false detection and missed detection caused by point cloud problems, thereby improving detection accuracy and providing more solid technical support for application scenarios such as autonomous driving and vehicle-road collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the overall workflow of the present invention;

[0071] Figure 2 This is a schematic diagram of the overall network framework of the present invention;

[0072] Figure 3 This is a schematic diagram of the density chiral sampling process of the present invention;

[0073] Figure 4 This is a basic flow chart of density chiral sampling of the present invention;

[0074] Figure 5 Schematic diagram of the basic process of positional chirality suppression processing in the present invention;

[0075] Figure 6 Schematic diagram of the workflow of the positional chirality suppression process of the present invention;

[0076] Figure 7 Schematic diagram showing the comparison of BEV AP (BEV Average Percision) detection accuracy between the method of the present invention and other methods;

[0077] Figure 8 Comparison of the 3D mean Average Percision (mAP) average detection accuracy between the method of the present invention and other methods;

[0078] Figure 9 This is a graph showing changes in the relevant indicators of the present invention with training rounds; DETAILED DESCRIPTION

[0079] It should be noted that:

[0080] Density chirality is defined as the difference in spatial distribution density due to different distances for virtual points generated from an image.

[0081] Position chirality is defined as the misalignment of points projected onto a symmetrical object due to factors such as depth completion noise, occlusion, or fusion errors.

[0082] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0083] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0084] Example 1

[0085] like Figures 1 and 2 An embodiment shown in FIG. 1 provides a multimodal 3D object detection method based on chirality perception fusion, including:

[0086] Step 1: Obtain the initial point cloud data and image data of the target to be measured;

[0087] Step 2: Input the point cloud data and image data into the trained target detection model for detection processing to obtain the detection results, which includes:

[0088] Step 2.1: Convert the two-dimensional pixel coordinates in the initial image data of the target to be measured into virtual points in three-dimensional space through the monocular depth estimation network and the virtual point generation network. Specifically:

[0089] Use the monocular depth estimation network to predict the image depth and obtain the predicted depth corresponding to each pixel coordinate in the initial image data;

[0090] According to the predicted depth and pixel coordinates, coordinate transformation is performed by obtaining the camera intrinsic parameters and extrinsic parameters of the image data to convert the two-dimensional coordinates of the pixels in the image data into the three-dimensional space coordinates of the virtual point;

[0091] Step 2.2: If Figures 3 and 4 As shown, the virtual points are randomly divided into a number of virtual point sets, and density chirality sampling is performed on the virtual points in the virtual point sets through a preset density chirality sampling module, specifically including:

[0092] Input the geometric gradient of the virtual point set, quantify the density chirality characteristics, and calculate the density chirality characteristics of the virtual point. The expression is:

[0093] ;

[0094] ;

[0095] in, is a virtual point set containing virtual point i and its neighborhood virtual point j, is the convolution kernel weight at the neighborhood point j, is the gradient amplitude of the neighborhood virtual point j, is the density chirality characteristic at the virtual point i; Density chirality Normalized density numerical expression; is the density chirality characteristic at the virtual point t, represents the natural exponential function, N is the number of virtual point sets;

[0096] Dynamic partitioning and importance sampling of virtual point sets: according to the density value of virtual point i Sort the virtual point set in descending order, and put the descending order before The first virtual point set is a dense area, and the remaining virtual point sets are sparse areas;

[0097] Sampling virtual point sets and retaining virtual point sets in dense areas The M virtual points with the highest internal density values ​​retain the virtual point set in the sparse area All virtual points in the, expression is:

[0098] ;

[0099] ;

[0100] in, represents the concatenation operation of a set of virtual points, A set of virtual points reserved for dense areas, is the virtual point set reserved for the sparse area, and M is the virtual point set in the dense area The number of virtual points retained in is the i-th virtual point set in the dense area, is the i-th virtual point set in the sparse area, is the separation threshold.

[0101] Where: the separation threshold The calculation method is:

[0102] ;

[0103] in, is the density value corresponding to the virtual point i, The function is used to find the maximum k value that satisfies the inequality, where k is a hyperparameter and τ is the chiral density accumulation threshold.

[0104] Step 2.3: Fuse the virtual points after density chirality sampling with the initial point cloud data, specifically including:

[0105] ;

[0106] F is the fused point cloud data, A set of virtual points reserved for dense areas, The virtual point set reserved for the sparse area, L is the point in the initial point cloud data.

[0107] Step 2.4: If Figure 5 As shown, the fused point cloud data is voxelized to obtain voxel units (chiral targets), specifically including:

[0108] The fused point cloud data is divided into voxel units of 0.1m×0.1m×0.2m, and the expression is:

[0109] ;

[0110] Where: T is the voxel unit, It is a voxelization operation.

[0111] Step 2.5: Perform position chirality suppression on the voxel unit and detect the data after position chirality suppression, specifically including:

[0112] Step 2.51: For each voxel unit T, define its local neighborhood, calculate the symmetry difference, and extract the position chirality feature , the expression is:

[0113] ;

[0114] in, is the chirality feature of the mth voxel unit position, is the neighboring voxel of the mth voxel unit Entropy of the distribution; is the balance coefficient, which is used to control the weight of entropy. is the neighboring voxel of the mth voxel unit The coordinates of Indicates the calculation of neighborhood voxel coordinates Its symmetrical voxel coordinates relative distance;

[0115] Among them, the voxel coordinates The specific calculation method is: learn the symmetry plane parameters through a lightweight multi-layer perceptron and perform adaptive symmetry plane transformation. The expression is:

[0116] ;

[0117] in, is the rotation matrix; is the translation vector;

[0118] Step 2.52: Obtain the 3D geometric features and 2D image features in the initial point cloud data and image data respectively through the virtual convolutional network;

[0119] Step 2.53: Use the attention mechanism to dynamically weight the 3D geometric features, 2D image features, and chirality features, and detect the dynamic weighted fusion features. The expression is:

[0120] ;

[0121] ;

[0122] in: is the weighted fusion feature, is the 3D geometric feature of the voxel unit containing the virtual point n, is the 2D image feature corresponding to the 3D geometric feature, Represents a learnable weight matrix with dimensions , represents the adaptive weight coefficient, is the Sigmoid function, is a learnable parameter, Represents a feature concatenation operation.

[0123] like Figure 6 As shown, this embodiment includes four different-sized position chirality suppression processes. In each position chirality suppression process, chiral feature extraction, adaptive symmetry plane learning, and feature fusion operations are performed in sequence.

[0124] Among them, the chiral feature extraction is performed in the following order: 1× downsampling step chiral feature extraction, 2× downsampling step chiral feature extraction, 4× downsampling step chiral feature extraction and 8× downsampling step chiral feature extraction.

[0125] Step 2.6: Detect the dynamic weighted fusion features, specifically including: generating proposals through the region proposal network, extracting the features of the region of interest, and then processing them through the detection head to finally obtain the detection results.

[0126] The comparison of 3D AP (3D Average Percision) detection accuracy between the method of the present invention and other methods is shown in Table 1:

[0127] Table 1 Performance comparison of the proposed method on the KITTI dataset test benchmark (3D AP)

[0128]

[0129] The data in Table 1 show that in terms of the 3D AP index of the KITTI dataset test benchmark, the method of the present invention shows outstanding advantages in different difficulty categories. In terms of the Easy 3D AP index, the method of the present invention reaches 95.20%, which is higher than Graph-VoI's 91.89% and BFT3D's 92.32%; the Mod 3D AP index is 87.48%, surpassing VirConv-S's 87.20%; and the Hard 3D AP index reaches 85.40%, which is significantly higher than UDeerPEP's 82.57% and MPCF's 80.69%. Compared with other methods, the method of the present invention is better in the 3D AP index values ​​of the three difficulty levels of Easy, Mod, and Hard, fully demonstrating the superiority of the method of the present invention in the 3DAP index.

[0130] like Figure 7 As shown in the figure, the BEV AP (Bird's Eye View Average Precision) indicator comparison of different methods at different difficulty levels (Easy, Mod, Hard):

[0131] As can be seen, the proposed method achieves high Easy BEV AP (orange line) values, outperforming all compared methods. For example, compared to methods such as Focals Conv and VPFNet, the proposed method achieves significantly higher Easy BEV AP values, demonstrating its superiority on simple tasks. The proposed method also achieves outstanding Mod BEV AP (red line), reaching a high percentage. Methods such as VirConv-L and Graph-VoI perform lower on this metric than the proposed method, demonstrating its good performance on moderately difficult tasks. The proposed method also performs well on Hard BEV AP (blue line), outperforming many compared methods. For example, methods such as Viklenet-R and VirConv-L achieve significantly lower Hard BEV AP values ​​than the proposed method, demonstrating its advantage on difficult tasks. Overall, the proposed method outperforms most compared methods on the Easy, Mod, and Hard BEV AP metrics, fully demonstrating its superiority in BEV AP.

[0132] like Figure 8 The figure shows a comparison of the 3D mAP (%) indicators of different methods, with the horizontal axis representing the method name and the vertical axis representing the 3D mAP value. The comparison shows that the value of the proposed method (87.99%) is higher than that of all other methods in terms of the 3D mAP indicator, such as VirConv-S (87.38%) and VirConv-T (86.68%), fully demonstrating the superiority of the proposed method in terms of the 3D mAP indicator.

[0133] In this embodiment, the training process of the target detection model specifically includes:

[0134] Obtain historical point cloud data, image data and corresponding target annotation information;

[0135] Construct a training set based on historical point cloud data, image data, and corresponding target annotation information, and divide the training data into several batches;

[0136] Input the training set into the target detection model in batches for training;

[0137] During the training process, the parameters of the target detection model are adjusted through the optimization algorithm to minimize the difference between the predicted results and the actual target annotation information;

[0138] After multiple rounds of iterative training, until the target detection model converges, a trained target detection model is obtained.

[0139] Among them: Figure 9 The following are the changes of different indicators with the training rounds (Epochs):

[0140] Easy 3D AP gradually increases with epochs, with significant improvements in the later stages (epochs 55-60), indicating that increasing training epochs has a positive impact on 3D AP for easier tasks. The Moderate 3D AP curve fluctuates but generally remains high, demonstrating the model's stable performance on moderately difficult tasks. Hard 3D AP fluctuates but stabilizes at a higher value in the later stages, indicating that the model's ability to handle difficult tasks is gradually improving with training. 3D mAP values ​​fluctuate between 87% and 89.5%, with a significant upward trend in the later stages, reflecting the model's overall 3D average precision improving with training. Each metric in the BEV AP series (Easy / Moderate / Hard / BEV mAP) shows a fluctuating upward trend. For example, Easy BEV AP approaches 96.4%, demonstrating that the model's performance from a bird's-eye view perspective gradually improves with each epoch, with improvements across tasks of varying difficulty. Test time fluctuates between 106 and 120 ms, indicating that test time varies somewhat per epoch, but remains within a reasonable range overall, without significant fluctuations or abnormal increases. Overall, most performance indicators (3D AP, BEV AP, mAP, etc.) increase or remain at a high level with fluctuations as the number of training rounds increases, reflecting the positive impact of training rounds on model performance; although the test time fluctuates, it is within an acceptable range and does not have a significant negative impact on model efficiency; the method of the present invention can be superimposed with the number of training rounds, and has positive trends and overall stability.

[0141] Example 2:

[0142] In this embodiment, the solution of the above embodiment 1 is applied to the autonomous driving scenario.

[0143] On urban roads at night, autonomous vehicles must detect dynamic obstacles such as pedestrians crossing the road and vehicles changing lanes. LiDAR detection suffers from sparse point clouds due to reflections from rain and fog, while cameras experience image noise due to low light conditions. Fusion algorithms are needed to improve the robustness of object detection.

[0144] Implementation steps:

[0145] First, data preprocessing and calibration are performed. The lidar point cloud is statistically filtered to remove outliers (such as rain and fog reflection noise). Based on the calibration matrix, the point cloud is projected into the camera coordinate system to achieve spatiotemporal synchronization. The camera image uses ISP dynamic parameter adjustment (such as white balance and dehazing algorithms) to optimize low-light image quality. After alignment with the lidar data, an RGB-D map is generated.

[0146] Next, virtual point generation and density sampling are performed. Virtual point generation is achieved by predicting image depth using a monocular depth estimation network (or PENet) such as Monodepth2, and generating a virtual point cloud to supplement sparse LiDAR areas (such as distant pedestrian outlines).

[0147] Next, density chirality sampling is performed. First, dynamic partitioning is used: the road area is divided into dense areas (near lane markings) and sparse areas (shoulders and sidewalks) based on a point cloud density threshold (τ = 0.8). Second, importance sampling is performed: virtual points with reflection intensities greater than 50% are retained in dense areas, while the entire point cloud is retained in sparse areas to ensure the integrity of pedestrian outlines. Third, voxelization and feature extraction are performed. Voxel division uses 0.1m×0.1m×0.2m voxel units, and the mean reflection intensity is calculated to distinguish between metal vehicles (mean > 120) and non-metallic obstacles (such as pedestrian backpacks). Fourth, position chirality suppression is performed: a four-layer suppression module is used to extract symmetric plane features (such as vehicle chassis symmetry) to suppress image noise caused by multipath reflections.

[0148] Finally, target detection and parameter optimization are performed. The fused voxel features are fed into the detection model, which outputs a 3D bounding box and class probabilities. Note that for rainy and foggy scenes, the balance coefficient can be adjusted from 0.5 to 0.7, prioritizing pedestrian detection recall and keeping the false alarm rate below 5%.

[0149] It should be noted that in autonomous driving scenarios, the voxel size of urban curves needs to be reduced to 0.05m to improve guardrail detection accuracy. In tunnel environments, the use of virtual point generation modules can be reduced as needed (depending on visual failure, the brightness and contrast of the image may need to be adjusted first).

[0150] Example 3:

[0151] In this embodiment, the above-mentioned surface autonomous navigation scenario in Example 1 is as follows:

[0152] In narrow inland waterways, unmanned boats need to avoid floating objects, other ships and shore obstacles, while also dealing with the problem of missing lidar point clouds caused by reflections from the water surface.

[0153] The specific implementation steps are:

[0154] First, multi-sensor data fusion preprocessing is performed. LiDAR point clouds can be combined with millimeter-wave radar data (which is rain and fog resistant) to remove water surface reflection noise. GNSS / IMU fusion is then used to calibrate the UAV's position. Polarization filters are used to suppress water surface reflections on camera images, and the BEV's perspective is converted to generate a 360° surround view segmentation map of the drivable area.

[0155] Next, virtual points are generated and dynamically partitioned to achieve density chiral sampling. A semantic segmentation network (such as DeepLabv3+) is used to identify the outlines of floating objects on the water surface, generating a virtual point cloud to fill in radar blind spots (such as transparent plastic bottles). τ = 0.6 can be adjusted based on the channel width (<20m), with a 3m range near the shore designated as a sparse zone, retaining all virtual points to prevent grounding. For voxelization and anti-interference processing, voxelization uses 0.3m × 0.3m × 0.5m voxel units (adapted to vessel size), and point density is used to identify dense clusters of floating objects (density >15 points / voxel). Furthermore, shoreline geometric constraints are introduced to suppress dynamic noise caused by waves, and point cloud data is stored using octree compression (saving 30% memory).

[0156] Finally, multi-target detection and obstacle avoidance decisions are performed. In addition to the 3D target detection used in this invention, YOLO-BEV can also be used to detect targets such as ships and buoys, combined with Kalman filtering to predict motion trajectories. For obstacle avoidance strategies, detour paths can be generated based on COLREGs rules. A "safe corridor" strategy is used for stationary obstacles, while the speed obstacle method is used to calculate avoidance maneuvers for dynamic targets.

[0157] It should be noted that during surface navigation, the voxel size can be increased to 0.5m in wide waters (>100m) to improve computational efficiency. In strong wind and wave environments, the IMU weight needs to be increased to 0.8 to stabilize pose estimation.

[0158] Example 4

[0159] This embodiment provides a multimodal 3D object detection device based on chirality perception fusion, including:

[0160] An acquisition module is used to obtain the initial point cloud data and image data of the target to be measured;

[0161] The processing module is used to input the point cloud data and image data into the trained target detection model for detection processing and obtain the detection results;

[0162] The process of performing detection processing on the trained target detection model to obtain the detection results specifically includes:

[0163] The pixel coordinates and predicted depth in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a virtual point generation network;

[0164] Randomly dividing the virtual points into a plurality of virtual point sets, and performing density chirality sampling on the virtual points in the virtual point sets;

[0165] Fuse the virtual points after density chirality sampling with the initial point cloud data;

[0166] The fused point cloud data is voxelized to obtain voxel units;

[0167] The voxel unit is subjected to position chirality suppression processing, and the data after the position chirality suppression processing is detected to obtain a detection result.

[0168] Example 5

[0169] This embodiment provides a multimodal 3D object detection system based on chirality perception fusion, including:

[0170] Memory, used to store computer programs / instructions;

[0171] A processor is used to execute the computer program / instructions to implement the steps of the above-mentioned embodiment 1.

[0172] In summary, the present invention introduces the concept of "chirality," i.e., the asymmetric characteristics of objects in three-dimensional space. This can enhance the model's understanding of the structural symmetry and directionality of objects (such as left and right turns of vehicles and differences in pedestrian postures), thereby more effectively suppressing noise and improving detection accuracy.

[0173] From the perspective of suppressing noise interference and improving detection accuracy, the application of "density chirality" and "position chirality" theories enables the model to more effectively handle issues such as uneven virtual point distribution and positional deviation. In complex scenarios, such as those with occlusion and noise interference, the chiral perception fusion mechanism employed by this invention can effectively reduce false detections and missed detections caused by point cloud issues, compared to traditional methods, thereby improving detection accuracy and providing more solid technical support for application scenarios such as autonomous driving and vehicle-infrastructure collaboration.

[0174] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A multimodal 3D object detection method based on chirality perception fusion, characterized in that: include: Obtain the initial point cloud data and image data of the target to be measured; Input the point cloud data and image data into the trained target detection model for detection processing to obtain the detection results; The process of performing detection processing on the trained target detection model to obtain the detection results specifically includes: The two-dimensional coordinates of pixels in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network; Randomly dividing the virtual points into a plurality of virtual point sets, and performing density chirality sampling on the virtual points in the virtual point sets; Fuse the virtual points after density chirality sampling with the initial point cloud data; The fused point cloud data is voxelized to obtain voxel units; The voxel unit is subjected to position chirality suppression processing, and target detection is performed on the data after the position chirality suppression processing to obtain a detection result.

2. The multimodal 3D object detection method based on chirality perception fusion according to claim 1, characterized in that: The method of converting the two-dimensional coordinates of pixels in the initial image data of the target to be measured into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network specifically includes: Use the monocular depth estimation network to predict the image depth and obtain the predicted depth corresponding to the two-dimensional coordinates of each pixel in the initial image data; According to the predicted depth, the two-dimensional coordinates of the pixels in the image data are converted into the three-dimensional space coordinates of the virtual point.

3. The multimodal 3D object detection method based on chirality perception fusion according to claim 2, characterized in that: The density chirality sampling of the virtual points in the virtual point set specifically includes: Calculate the density chirality characteristics of the virtual point, the expression is: ； ； in, is a virtual point set containing virtual point i and its neighborhood virtual point j, is the convolution kernel weight at the neighborhood point j, is the gradient amplitude of the neighborhood virtual point j, is the density chirality characteristic at the virtual point i; Density chirality Normalized density numerical expression; is the density chirality characteristic at the virtual point t, represents the natural exponential function, N is the number of virtual point sets; According to the density value of virtual point i Sort the virtual point set in descending order, and put the descending order before The first virtual point set is a dense area, and the remaining virtual point sets are sparse areas; Step 3: Sample the virtual point set and retain the virtual point set in the dense area The M virtual points with the highest internal density values ​​retain the virtual point set in the sparse area All virtual points in the, expression is: ； ； in, represents the concatenation operation of a set of virtual points, The sum of the virtual point sets reserved for dense areas, is the sum of the virtual point sets reserved for the sparse area, and M is the virtual point set in the dense area The number of virtual points retained in is the i-th virtual point set in the dense area, is the i-th virtual point set in the sparse area, is the separation threshold.

4. The multimodal 3D object detection method based on chirality perception fusion according to claim 3, characterized in that: The separation threshold The calculation method is: ； in, is the density value corresponding to the virtual point i, The function is used to find the maximum k value that satisfies the inequality, where k is a hyperparameter and τ is the chiral density accumulation threshold.

5. The multimodal 3D object detection method based on chirality perception fusion according to claim 3, characterized in that: The fusion processing of the virtual points after density chirality sampling with the initial point cloud data specifically includes: ； F is the fused point cloud data, The sum of the virtual point sets reserved for dense areas, The sum of the virtual point sets retained for the sparse area, L is the point in the initial point cloud data.

6. The multimodal 3D object detection method based on chirality perception fusion according to claim 5, characterized in that: The fused point cloud data is voxelized to obtain voxel units, which includes: The fused point cloud data is divided into voxel units of fixed size, and the expression is: ； Where: T is the voxel unit, It is a voxelization operation.

7. The multimodal 3D object detection method based on chirality perception fusion according to claim 6, characterized in that: The performing position chirality suppression processing on the voxel unit, performing target detection on the data after the position chirality suppression processing, and obtaining the detection result specifically includes: For each voxel unit T, define its local neighborhood, calculate the symmetry difference, and extract the position chirality feature , the expression is: ； in, is the chirality feature of the mth voxel unit position, is the neighboring voxel of the mth voxel unit Entropy of the distribution; is the balance coefficient, which is used to control the weight of entropy. is the neighboring voxel of the mth voxel unit The coordinates of Indicates the calculation of neighborhood voxel coordinates Its symmetrical voxel coordinates relative distance; Among them, the voxel coordinates The specific calculation method is: learn the symmetry plane parameters through a lightweight multi-layer perceptron and perform adaptive symmetry plane transformation. The expression is: ； in, is the rotation matrix; is the translation vector; The 3D geometric features and 2D image features in the initial point cloud data and image data are obtained through the virtual convolutional network respectively; The 3D geometric features, 2D image features, and chirality features are dynamically weighted and fused through the attention mechanism, and the dynamic weighted fusion features are detected. The expression is: ； ； in: is the dynamic weighted fusion feature, is the 3D geometric feature of the voxel unit containing the virtual point n, is the 2D image feature corresponding to the 3D geometric feature, Represents a learnable weight matrix with dimensions , represents the adaptive weight coefficient, is the Sigmoid function, is a learnable parameter, Represents feature concatenation operation; The obtained dynamic weighted fusion features are used to generate proposals through the region proposal network, extract the features of the region of interest, and then processed by the detection head to finally obtain the target detection results.

8. The multimodal 3D object detection method based on chirality perception fusion according to claim 1, characterized in that: The training process of the target detection model specifically includes: Obtain historical point cloud data, image data and corresponding target annotation information; Construct a training set based on historical point cloud data, image data, and corresponding target annotation information, and divide the training data into several batches; Input the training set into the target detection model in batches for training; During the training process, the parameters of the target detection model are adjusted through the optimization algorithm to minimize the difference between the predicted results and the actual target annotation information; After multiple rounds of iterative training, until the target detection model converges, a trained target detection model is obtained.

9. A multimodal 3D object detection device based on chirality perception fusion, characterized in that: include: An acquisition module is used to obtain the initial point cloud data and image data of the target to be measured; The processing module is used to input the point cloud data and image data into the trained target detection model for detection processing and obtain the detection results; The process of performing detection processing on the trained target detection model to obtain the detection results specifically includes: The two-dimensional coordinates of pixels in the initial image data of the target to be measured are converted into virtual points in three-dimensional space through a monocular depth estimation network and a virtual point generation network; Randomly dividing the virtual points into a plurality of virtual point sets, and performing density chirality sampling on the virtual points in the virtual point sets; Fuse the virtual points after density chirality sampling with the initial point cloud data; The fused point cloud data is voxelized to obtain voxel units; The voxel unit is subjected to position chirality suppression processing, and target detection is performed on the data after the position chirality suppression processing to obtain a detection result.

10. A multimodal 3D object detection system based on chirality perception fusion, characterized in that: include: Memory, used to store computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of the multimodal 3D target detection method based on chirality perception fusion according to any one of claims 1 to 8.

Images