Multi-modal data three-dimensional target detection method, system, device and medium based on image depth estimation and point cloud fusion

Abstract

The application provides a kind of multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, by the initial image data of target monocular depth estimation and semantic segmentation, obtain per-pixel depth data and foreground pixel data;Then, per-pixel depth data is restored into pseudo point cloud, and the pseudo point cloud is classified in combination with the foreground pixel, to obtain pseudo point cloud classification information;Then, the initial point cloud data of target is acquired, and the pseudo point cloud data is fused to obtain fusion point cloud data;Then, the fusion point cloud data is subjected to semantic refinement processing, voxelization processing, feature aggregation processing, to obtain fusion voxel high-dimensional feature;Then, the fusion fusion voxel high-dimensional feature is subjected to feature convolution operation, to obtain two-dimensional feature map;Finally, the two-dimensional feature map is subjected to sliding window and anchor box operation, to realize efficient and accurate target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing and point cloud data processing, and particularly to a multi-modal data three-dimensional target detection method, system, device and medium based on image depth estimation and point cloud fusion. BACKGROUND

[0002] Three-dimensional target detection is an important task in the field of computer vision, aiming to detect and locate objects from three-dimensional scenes. Unlike traditional two-dimensional target detection, three-dimensional target detection not only focuses on the position of objects in the image plane, but also requires estimating the position, pose and size of objects in three-dimensional space. This makes three-dimensional target detection have more extensive practical use in many applications, such as autonomous driving, robot navigation, augmented reality, etc.

[0003] Single-modal three-dimensional target detection refers to using only one kind of sensor information, such as using only lidar or only camera. The limitations of single sensor may lead to insufficient environmental information, making it difficult to accurately capture three-dimensional information in certain scenes or conditions. Moreover, single-modal three-dimensional target detection is more dependent on the characteristics of the environment, and is more sensitive to environmental factors such as light, shadow and occlusion, limiting its robustness in complex and variable environments. Multi-modal three-dimensional target detection refers to using information from different sensors, fully combining data from multiple sensors such as lidar and camera, to comprehensively and accurately understand three-dimensional scenes and improve the accuracy of three-dimensional target detection. Among them, the camera can provide rich object appearance features and can better complete target detection and classification, and the point cloud can provide accurate geometric features and spatial information of objects and can better complete the positioning of targets in three-dimensional space. However, existing multi-modal three-dimensional target detection techniques often focus on the data fusion level, and do not fully utilize the feature data of single modal in the fusion process, resulting in a certain degree of information waste, and further leading to the complexity of the existing method in the data fusion level algorithm, and the poor data processing efficiency and real-time performance. SUMMARY

[0004] Aiming at the problems of insufficient mining of single modal information, low utilization rate and complex data fusion level algorithm in the prior art, the present application provides a multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, and a system, device and medium applying the method. The detection method performs monocular depth estimation and semantic segmentation on the initial image data of the target to obtain pixel-by-pixel depth data and foreground pixel data; then the pixel-by-pixel depth data is restored into pseudo point cloud, and the pseudo point cloud is classified in combination with the foreground pixels to obtain pseudo point cloud classification information; then the initial point cloud data of the target is obtained and fused with the pseudo point cloud data to obtain fused point cloud data; then the fused point cloud data is subjected to semantic refinement processing, voxelization processing and feature aggregation processing to obtain fused voxel high-dimensional features; then the fused fused voxel high-dimensional features are subjected to feature convolution operation to obtain a two-dimensional feature map; finally, the two-dimensional feature map is subjected to sliding window and anchor box placement operation to realize efficient and accurate target detection.

[0005] The technical scheme of the present application is:

[0006] The multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion comprises the following steps:

[0007] Step 1: obtaining initial image data of a target; performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data, and performing semantic segmentation on the initial image data to obtain foreground pixel data;

[0008] Step 2: converting each pixel in the initial image from a plane coordinate to a three-dimensional coordinate using the pixel-by-pixel depth data obtained in step 1 to obtain pseudo point cloud data of all pixels; and classifying the pseudo point cloud data according to the foreground pixel data obtained in step 1 to obtain pseudo point cloud classification information, and adding the classification information as additional features to the pseudo point cloud data;

[0009] Step 3: obtaining initial point cloud data of the target, performing information statistics on the pseudo point cloud data and the initial point cloud data, selecting all the initial point cloud data of the target within a range of 0-20 meters, and selecting 10% of the pseudo point cloud data in a farthest point sampling manner, selecting all the pseudo point cloud data within a range greater than 20 meters, and selecting 10% of the target initial point cloud data in a farthest point sampling manner to obtain fused point cloud data;

[0010] Step 4: performing semantic refinement processing on the fusion point cloud data obtained in step 3: locating point cloud data carrying category information in the fusion point cloud, distributing the category information to adjacent points within a radius of a set distance k, then performing columnar voxel division processing on the fusion point cloud data to obtain fusion voxels, and performing feature aggregation processing to obtain high-dimensional features of the fusion voxels;

[0011] Step 5: performing feature convolution operation on the high-dimensional features of the fusion voxels obtained in step 4 to obtain a two-dimensional feature map;

[0012] Step 6: performing sliding window and anchor box placement operation on the two-dimensional feature map obtained in step 5 to obtain a target detection result.

[0013] Further, in step 1, the process of performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data is as follows:

[0014] Performing inverse projection matrix operation on the initial image data to obtain the plane coordinates of all pixels in the initial image in the world coordinate system, and generating a set of depth information for each pixel to obtain a candidate set of pixel depth information;

[0015] Performing normalization processing and linear processing on the candidate set of pixel depth information to obtain a set of depth feature vectors of all pixels;

[0016] Performing one-dimensional convolution, RELU activation function, and full connection layer on the set of depth feature vectors to obtain the pixel-by-pixel depth data.

[0017] Further, in step 1, the process of performing semantic segmentation on the initial image data to obtain foreground pixel data is as follows:

[0018] Performing normalization and scale adjustment operation on the initial image data to obtain image data of appropriate size;

[0019] Performing two-dimensional convolution and maximum pooling operation on the image data of appropriate size to obtain a high-level semantic feature map;

[0020] Performing pixel classification probability processing on the high-level semantic feature map to obtain a set of pixel-by-pixel category probability;

[0021] Extracting the maximum category probability from the set of pixel-by-pixel category probability, and taking the classification result corresponding to the maximum category probability as the pixel-by-pixel classification;

[0022] Performing category statistics on the obtained pixel-by-pixel classification information to obtain pixels belonging to the foreground classification as the foreground pixel data.

[0023] Further, in step 2, the process of performing ray-reduction operation on each pixel to obtain the pseudo point cloud data of all pixels is as follows:

[0024] According to the initial image data of the target, in combination with the internal and external parameter information of the camera, the initial image pixels are converted to the camera coordinate system to obtain the planar coordinates of the pixels in the camera coordinate system;

[0025] In combination with the pixel-by-pixel depth data obtained in step 1, the planar coordinates of the pixels in the camera coordinate system are converted to three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data.

[0026] Further, in step 2, the process of classifying the pseudo point cloud data to obtain the pseudo point cloud classification information is as follows:

[0027] According to the size of the initial image of the target, a blank tensor of the same size is created and initialized to 0;

[0028] According to the foreground pixel data obtained in step 1, the value at the position corresponding to the foreground pixel in the blank tensor is modified to 1 through a position index operation to obtain a foreground pixel mask;

[0029] According to the foreground pixel mask, the pseudo point cloud generated by the foreground pixels is obtained, which is marked as foreground pseudo point cloud, and the rest is marked as background pseudo point cloud to obtain the pseudo point cloud classification information, and the classification information is embedded into the pseudo point cloud; Specifically, the class information is attached to the information field, and the original information of each point has three-dimensional coordinates and reflectivity, and after embedding the class, each point has three-dimensional coordinates, reflectivity, and class information.

[0030] Further, in step 3, the process of performing distance division and fusion operation on the pseudo point cloud data and the initial point cloud data is as follows:

[0031] According to the range of the initial point cloud data, a step unit is selected to divide the distance of the pseudo point cloud data and the initial point cloud data to obtain a point cloud distance classification set;

[0032] According to the sampling number in the pre-defined step unit and the point cloud distance classification set, the pseudo point cloud data and the initial point cloud data are sampled, and then the sampled pseudo point cloud and the initial point cloud are merged to obtain the fusion point cloud data.

[0033] Further, in step 5, the process of performing feature convolution operation is as follows:

[0034] According to the pre-set voxel size and the fusion voxel high-dimensional feature, the fusion voxel high-dimensional feature is mapped to a plane by reducing the number of channels to obtain a plane feature map;

[0035] The plane feature map is subjected to two-dimensional convolution and pooling operation to obtain a two-dimensional feature map.

[0036] Further, the process of sliding window and placing anchor frame operation in step 6 is as follows:

[0037] Using a fixed size window, sliding on the two-dimensional feature map with a fixed stride; at each window position, placing several anchor frames of different scales as candidate regions; performing feature extraction on each candidate region, and inputting the extracted features into the target classifier to determine whether the target object is contained in the window; finally, integrating all the detection results in the window to obtain the final target detection result.

[0038] Based on the above method, the application further provides a multi-modal data three-dimensional target detection system, comprising

[0039] A pixel depth data generation module is configured to perform monocular depth estimation on the obtained target initial image data to obtain pixel-by-pixel depth information.

[0040] A foreground pixel generation module is configured to perform semantic segmentation on the obtained target initial image data to obtain foreground pixel data.

[0041] A pseudo point cloud classification information generation module is configured to perform a ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo point cloud data, and perform classification on the pseudo point cloud data using the foreground pixel data output by the foreground pixel generation module to obtain pseudo point cloud classification information.

[0042] A fused point cloud data generation module is configured to perform distance division on the pseudo point cloud data and the initial point cloud data to obtain a point cloud distance classification set, and fuse the pseudo point cloud data and the initial point cloud data within a set distance range based on the point cloud distance classification set to form the fused point cloud data.

[0043] A three-dimensional feature fusion module is configured to perform semantic refinement processing on the fused point cloud data and perform voxel division to obtain voxel data corresponding to all point clouds, and perform feature aggregation processing on the voxel data to obtain fused voxel high-dimensional features.

[0044] A two-dimensional feature map generation module is configured to perform feature convolution coding processing on the fused voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map.

[0045] A detection result generation module is configured to perform sliding window and anchor frame placement operation on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

[0046] In addition, the application further provides a computer device and a computer readable storage medium:

[0047] The computer device comprises a processor and a memory, and the processor realizes the above-mentioned multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion when executing the computer program stored in the memory.

[0048] The computer readable storage medium stores a computer program, and the computer program realizes the above-mentioned multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion when executed by the processor.

[0049] Advantages

[0050] The application provides a multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, which comprises the following steps: performing monocular depth estimation and semantic segmentation on initial image data of a target to obtain pixel-by-pixel depth data and foreground pixel data; then, restoring the pixel-by-pixel depth data into pseudo point cloud and classifying the pseudo point cloud in combination with the foreground pixel to obtain pseudo point cloud classification information; then, obtaining initial point cloud data of the target and performing fusion operation on the pseudo point cloud data to obtain fusion point cloud data; then, performing semantic refinement processing, voxelization processing and feature aggregation processing on the fusion point cloud data to obtain fusion voxel high-dimensional features; then, performing feature convolution operation on the fusion fusion voxel high-dimensional features to obtain a two-dimensional feature map; and finally, performing sliding window and anchor box operation on the two-dimensional feature map to realize efficient and accurate target detection.

[0051] The application provides a multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, which utilizes an image monocular depth estimation method to perform fusion processing on image pixels and real point cloud data, effectively solving the information waste problem existing in conventional multi-modal three-dimensional target detection schemes.

[0052] Additional aspects and advantages of the application will be described in part in the description which follows, and in part will become apparent to those skilled in the art from the description, or by practicing the application. BRIEF DESCRIPTION OF DRAWINGS

[0053] The above and / or additional aspects and advantages of the application will become apparent and be readily understood from the following description, taken in connection with the drawings, in which:

[0054] Figure 1 The method flowchart of the application. DETAILED DESCRIPTION

[0055] The embodiments of the application are described in detail below, which are exemplary and intended to explain the application, and cannot be understood as a limitation of the application.

[0056] The embodiment provides a multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, which comprises the following steps:

[0057] Step 1: obtaining initial image data of the target; monocular depth estimation is performed on the initial image data to obtain pixel-by-pixel depth data, and semantic segmentation is performed on the initial image data to obtain foreground pixel data.

[0058] In this embodiment, a camera device is used to obtain images in the range of the target to be detected, forming initial image data of the target. In order to improve the training efficiency of the model, data enhancement processing is performed on the initial image data. The data enhancement operation includes but is not limited to random cropping and random flipping.

[0059] Then monocular depth estimation is performed on the obtained initial image data of the target region to obtain pixel-by-pixel depth data.

[0060] According to the camera internal and external parameter matrix, inverse projection matrix operation is performed on the initial image data to convert the pixel coordinates to the camera coordinate system and then to the world coordinate system, obtaining the plane coordinates of all pixels in the initial image in the world coordinate system; a set of depth information is generated at a fixed interval distance for each pixel for candidate, obtaining a pixel depth information candidate set; the pixel depth information candidate set is processed by normalization, linear processing, and maximum pooling to predict the depth information, obtaining a depth feature vector set of all pixels, which is processed by one-dimensional convolution, RELU activation function, and full connection layer to retain the highest score depth information representing the pixel depth information, and then nonlinearly transformed to obtain the pixel-by-pixel depth data. After this step, the depth information corresponding to the image pixels is obtained.

[0061] Then semantic segmentation is performed on the obtained initial image data of the target region to obtain foreground pixel data.

[0062] Normalization and scale adjustment operations are performed on the initial image data to obtain image data of a proper size (370x1240); two-dimensional convolution and maximum pooling operations are performed on the image data of the proper size to obtain high-level semantic feature maps; pixel classification probability processing is performed on the high-level semantic feature maps to obtain a pixel-by-pixel class probability set; the maximum class probability value is extracted from the pixel-by-pixel class probability set, and the classification result corresponding to the maximum class probability value is taken as the pixel-by-pixel classification; class statistics is performed on the obtained pixel-by-pixel classification information to obtain pixels belonging to the foreground classification as the foreground pixel data.

[0063] Step 2: for the pixel-by-pixel depth data obtained in step 1, each pixel in the initial image is converted from plane coordinates to three-dimensional coordinates to obtain pseudo point cloud data of all pixels; and according to the foreground pixel data obtained in step 1, the pseudo point cloud data is classified to obtain pseudo point cloud classification information.

[0064] First, according to the initial image data of the target, combined with the camera internal and external parameter information, the initial image pixel is converted to the camera coordinate system to obtain the planar coordinates of the pixel in the camera coordinate system; then, combined with the pixel-by-pixel depth data obtained in step 1, the planar coordinates of the pixel in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data.

[0065] In order to distinguish the point cloud categories (such as vehicles and non-vehicles) in the pseudo point cloud data, the pseudo point cloud is classified according to the pseudo point cloud coordinate information and the foreground pixel data:

[0066] According to the size of the initial image of the target, a blank tensor of the same size is created and all initialized to 0;

[0067] According to the foreground pixel data obtained in step 1, the coordinate position thereof in the initial image is obtained, and a unique index is assigned to each foreground pixel;

[0068] According to the position index, the value of the corresponding foreground pixel position in the blank tensor is modified to 1 to obtain the foreground pixel mask;

[0069] According to the foreground pixel mask in the pseudo point cloud data space, the position index is obtained to generate the pseudo point cloud of the foreground (vehicle) pixel, which is marked as foreground pseudo point cloud, and the rest is marked as background pseudo point cloud to obtain the pseudo point cloud classification information.

[0070] Step 3: Obtain the initial point cloud data of the target, and perform information statistics on the pseudo point cloud data and the initial point cloud data, select all the initial point cloud of the target and 10% of the pseudo point cloud data in the farthest point sampling manner within the range of 0-20 meters, select all the pseudo point cloud data and 10% of the target initial point cloud data in the farthest point sampling manner within the range of more than 20 meters, to obtain the fusion point cloud data.

[0071] Use the laser radar to obtain the point cloud data in the range of the target to be detected as the initial point cloud data of the target region, and the point cloud data includes but is not limited to three-dimensional coordinate information and reflection intensity information.

[0072] According to the initial point cloud data, determine the nearest distance and the farthest distance in the point cloud, and perform information statistics on the pseudo point cloud data and the initial point cloud data to obtain a plurality of distance division intervals, and in each distance division interval, collect the point cloud contained in the distance range to form a point cloud distance classification set;

[0073] According to the sampling number in the pre-defined setting step unit and the point cloud distance classification set, the pseudo point cloud data and the initial point cloud data are sampled, and then the sampled pseudo point cloud and initial point cloud are merged to obtain the fusion point cloud data.

[0074] Step 4: The fusion point cloud data obtained in step 3 is subjected to semantic refinement processing, the point cloud carrying category information in the fusion point cloud is located, the category information is assigned to the neighboring points within a radius of k, and then the fusion point cloud data is subjected to columnar voxel division processing to obtain fusion voxels, and the fusion voxel high-dimensional features are obtained through feature aggregation processing.

[0075] In step 2, the classification information has been embedded into the pseudo point cloud. First, the point cloud carrying category information needs to be located in the fusion point cloud data.

[0076] Taking the point cloud carrying category information as a center point (which can be referred to as an information center point) and k as a radius, a set of neighboring points within the radius region is obtained, the point cloud data closest to the information center point is captured to obtain a set of point cloud nearest neighbors of the information center point. The classification information of the information center point is assigned to the corresponding point cloud nearest neighbors, and the association between the foreground pseudo point cloud and the actual initial point cloud is established, so that each point in the fusion point cloud carries classification information from its nearest neighbor pseudo point cloud.

[0077] Then, the fusion point cloud data is subjected to columnar voxel division processing to map the point cloud information to a regular three-dimensional voxel grid to obtain fusion voxels, and high-dimensional features of the fusion voxels are extracted to integrate the feature information of the point cloud in the voxels to obtain high-dimensional features of the fusion voxels.

[0078] Step 5: The fusion voxel high-dimensional features obtained in step 4 are subjected to feature convolution operation to obtain a two-dimensional feature map.

[0079] According to the preset voxel size and the fusion voxel high-dimensional features, the fusion voxel high-dimensional features are mapped to a plane by reducing the number of channels to obtain a plane feature map, and the plane feature map is subjected to two-dimensional convolution and pooling operation to obtain a two-dimensional feature map. Specifically, the plane feature map is obtained by:

[0080] The center of the fusion voxel is projected onto the horizontal plane to form a top-down view, the top-down view is divided into fixed-size lattices, each lattice corresponds to a region in the top-down view, the number of voxels in each lattice is counted and averaged to obtain a blank plane feature map.

[0081] The voxel high-dimensional features are filled into the blank plane feature map by using the space-to-channel method to obtain a plane feature map.

[0082] The plane feature map is subjected to two-dimensional convolution, maximum pooling, and nonlinear transformation to obtain a two-dimensional feature map.

[0083] Step 6: The two-dimensional feature map obtained in step 5 is subjected to sliding window and anchor box placement operation to obtain a target detection result.

[0084] Using a fixed-size window, slide on the two-dimensional feature map with a fixed stride to capture information at different positions in the image; at each window position, place several anchor boxes of different proportions as candidate regions; perform feature extraction on each candidate region, that is, extract the corresponding feature map region within each anchor box and obtain the feature representation related to the region through a convolutional neural network or other feature extraction method.

[0085] Input the extracted features into the target classifier to determine whether the target object is contained in the window; the pre-trained target classifier is usually a deep learning model, which can be a convolutional neural network (CNN) or other models suitable for classification tasks. The classifier evaluates each candidate region and outputs a probability or confidence score indicating whether the region contains a target object.

[0086] Finally, integrate all window detection results to obtain the final target detection result.

[0087] Based on the above method, the embodiment also proposes a multi-modal data three-dimensional target detection system, comprising:

[0088] A pixel depth data generation module is configured to perform monocular depth estimation on the obtained target initial image data to obtain pixel-by-pixel depth information.

[0089] A foreground pixel generation module is configured to perform semantic segmentation on the obtained target initial image data to obtain foreground pixel data.

[0090] A pseudo-point cloud classification information generation module is configured to perform a ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo-point cloud data, and classify the pseudo-point cloud data using the foreground pixel data output by the foreground pixel generation module to obtain pseudo-point cloud classification information.

[0091] A fusion point cloud data generation module is configured to divide the pseudo-point cloud data and the initial point cloud data by distance to obtain a point cloud distance classification set, and fuse the pseudo-point cloud data and the initial point cloud data within a set distance range based on the point cloud distance classification set to form the fusion point cloud data.

[0092] A three-dimensional feature fusion module is configured to perform semantic refinement processing on the fusion point cloud data and divide the data into voxels to obtain voxel data corresponding to all point clouds. The voxel data is processed by feature aggregation to obtain fusion voxel high-dimensional features.

[0093] A two-dimensional feature map generation module is configured to perform feature convolutional coding processing on the fusion voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map.

[0094] The detection result generation module is configured to perform sliding window and anchor box placement operations on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

[0095] In addition, the present application also provides a computer device and a computer readable storage medium:

[0096] The computer device comprises a processor and a memory, and the processor implements the above-mentioned multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion when executing the computer program stored in the memory.

[0097] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the above-mentioned multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion.

[0098] The multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion provided in the embodiment comprises the following steps: performing monocular depth estimation and semantic segmentation on initial image data of a target to obtain pixel-by-pixel depth data and foreground pixel data; then, restoring the pixel-by-pixel depth data into pseudo point cloud and classifying the pseudo point cloud in combination with the foreground pixel to obtain pseudo point cloud classification information; then, obtaining initial point cloud data of the target and performing fusion operation on the pseudo point cloud data to obtain fusion point cloud data; then, performing semantic refinement processing, voxelization processing and feature aggregation processing on the fusion point cloud data to obtain fusion voxel high-dimensional features; then, performing feature convolution operation on the fusion fusion voxel high-dimensional features to obtain a two-dimensional feature map; and finally, performing sliding window and anchor box placement operations on the two-dimensional feature map to realize efficient and accurate target detection.

[0099] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments without departing from the principles and purposes of the present application within the scope of the present application.

Claims

1. A method for multi-modal data three-dimensional object detection based on image depth estimation and point cloud fusion, characterized in that: The method comprises the following steps: Step 1: obtaining initial image data of a target; monocular depth estimation is performed on the initial image data to obtain pixel-by-pixel depth data, and semantic segmentation is performed on the initial image data to obtain foreground pixel data; Step 2: converting each pixel in the initial image from a plane coordinate to a three-dimensional coordinate by using the pixel-by-pixel depth data obtained in step 1 to obtain pseudo point cloud data of all pixels; and classifying the pseudo point cloud data according to the foreground pixel data obtained in step 1 to obtain pseudo point cloud classification information, and adding the class information as additional features to the pseudo point cloud data; Step 3: obtaining initial point cloud data of the target, performing information statistics on the pseudo point cloud data and the initial point cloud data, and selecting all initial point cloud data of the target within a range of 0-20 meters, and selecting 10% of the pseudo point cloud data in a farthest point sampling manner, and selecting all pseudo point cloud data within a range greater than 20 meters, and selecting 10% of the initial point cloud data of the target in a farthest point sampling manner to obtain fused point cloud data; Step 4: performing semantic refinement processing on the fused point cloud data obtained in step 3: locating the point cloud data carrying the class information in the fused point cloud, distributing the class information to the neighboring points within a radius of a set distance k, and then performing columnar voxel division processing on the fused point cloud data to obtain fused voxels, and performing feature aggregation processing to obtain high-dimensional features of the fused voxels; Step 5: performing feature convolution operation on the high-dimensional features of the fused voxels obtained in step 4 to obtain a two-dimensional feature map; Step 6: performing sliding window and anchor box placement operation on the two-dimensional feature map obtained in step 5 to obtain a target detection result. 2.The method of claim 1, wherein: In step 1, the process of performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data is as follows: performing inverse projection matrix operation on the initial image data to obtain the plane coordinates of all pixels in the initial image under the world coordinate system, and generating a set of depth information for each pixel to obtain a candidate set of pixel depth information; performing normalization processing and linear processing on the candidate set of pixel depth information to obtain a set of depth feature vectors of all pixels; performing one-dimensional convolution, RELU activation function and full connection layer on the set of depth feature vectors to obtain the pixel-by-pixel depth data. 3.The method of claim 1, wherein: In step 1, the process of performing semantic segmentation on the initial image data to obtain foreground pixel data is as follows: performing normalization and scale adjustment operation on the initial image data to obtain image data of appropriate size; performing two-dimensional convolution and maximum pooling operation on the image data of appropriate size to obtain a high-level semantic feature map; performing pixel classification probability processing on the high-level semantic feature map to obtain a set of pixel-by-pixel class probability; extracting the maximum class probability from the set of pixel-by-pixel class probability, and taking the classification result corresponding to the maximum class probability as the pixel-by-pixel classification; performing class statistics on the obtained pixel-by-pixel classification information to obtain pixels belonging to the foreground classification as the foreground pixel data.

4. The method of claim 1, wherein: In step 2, the process of performing ray restoration operation on each pixel to obtain pseudo point cloud data of all pixels is as follows: According to the initial image data of the target, combined with the camera internal and external parameter information, the initial image pixels are converted to the camera coordinate system to obtain the planar coordinates of the pixels in the camera coordinate system; Combined with the pixel-by-pixel depth data obtained in step 1, the planar coordinates of the pixels in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data; Further, in step 2, the pseudo point cloud data is classified to obtain the pseudo point cloud classification information, and the process is as follows: According to the size of the initial image of the target, a blank tensor of the same size is created and all are initialized to 0; According to the foreground pixel data obtained in step 1, the value at the position corresponding to the foreground pixel in the blank tensor is modified to 1 through a position index operation to obtain a foreground pixel mask; According to the foreground pixel mask, the pseudo point cloud generated by the foreground pixels is obtained, which is marked as foreground pseudo point cloud, and the rest is marked as background pseudo point cloud to obtain the pseudo point cloud classification information, and the classification information is embedded into the pseudo point cloud; Specifically, the class information is attached to the information field, and each point has three-dimensional coordinates and reflectivity. After embedding the class, each point has three-dimensional coordinates, reflectivity, and class information.

5. The method of claim 1, wherein: In step 3, the process of distance division and fusion operation of the pseudo point cloud data and the initial point cloud data is as follows: According to the range of the initial point cloud data, a step unit is selected to divide the distance of the pseudo point cloud data and the initial point cloud data to obtain a point cloud distance classification set; According to the sampling number in the pre-defined step unit and the point cloud distance classification set, the pseudo point cloud data and the initial point cloud data are sampled, and then the sampled pseudo point cloud and the initial point cloud are merged to obtain the fusion point cloud data.

6. The method of claim 1, wherein: In step 5, the process of feature convolution operation is as follows: According to the preset voxel size and the fusion voxel high-dimensional feature, the fusion voxel high-dimensional feature is mapped to a plane by reducing the number of channels to obtain a plane feature map; The plane feature map is subjected to two-dimensional convolution and pooling operation to obtain a two-dimensional feature map.

7. The method of claim 1, wherein: In step 6, the process of sliding window and anchor box placement operation is as follows: A fixed size window is used to slide on the two-dimensional feature map with a fixed step size; in each window position, several anchor boxes of different proportions are placed as candidate regions; feature extraction is performed on each candidate region, and the extracted features are input into a target classifier to determine whether the target object is contained in the window; finally, the detection results in all windows are integrated to obtain the final target detection result.

8. A multi-modal data three-dimensional object detection system, comprising: The method comprises A pixel depth data generation module for monocular depth estimation on the obtained target initial image data to obtain pixel-by-pixel depth information; A foreground pixel generation module for semantic segmentation on the obtained target initial image data to obtain foreground pixel data; A pseudo point cloud classification information generation module for ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo point cloud data, and classifying the pseudo point cloud data by using the foreground pixel data output by the foreground pixel generation module to obtain pseudo point cloud classification information; The fusion point cloud data generation module is configured to perform distance division on the pseudo point cloud data and the initial point cloud data to obtain a point cloud distance classification set, and fuse the pseudo point cloud data and the initial point cloud data within a set distance range based on the point cloud distance classification set to form the fusion point cloud data. The three-dimensional feature fusion module is configured to perform semantic refinement processing on the fusion point cloud data, and perform voxel division to obtain voxel data corresponding to all point clouds, and perform feature aggregation processing on the voxel data to obtain fusion voxel high-dimensional features. The two-dimensional feature map generation module is configured to perform feature convolution coding processing on the fusion voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map. The detection result generation module is configured to perform sliding window and anchor box placement operations on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

9. A computer device, comprising: The computer device includes a processor and a memory, and the processor implements the method of any one of claims 1-7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

Images