Multi-modal data three-dimensional target detection method, system and equipment based on image depth estimation and point cloud fusion, and medium

Abstract

The invention provides a multi-modal data three-dimensional target detection method based on image depth estimation and point cloud fusion, and the method comprises the steps: carrying out the monocular depth estimation and semantic segmentation of the initial image data of a target, and obtaining the per-pixel depth data and foreground pixel data; thirdly, restoring the pixel-by-pixel depth data into false point clouds, and classifying the false point clouds in combination with foreground pixels to obtain false point cloud classification information; then, acquiring initial point cloud data of a target, and performing fusion operation on the initial point cloud data and the false point cloud data to obtain fused point cloud data; then, semantic refinement processing, voxel division processing and feature aggregation processing are carried out on the fused point cloud data, and fused voxel high-dimensional features are obtained; then, performing feature convolution operation on the fused voxel high-dimensional features to obtain a two-dimensional feature map; and finally, performing window sliding and anchor frame placing operations on the two-dimensional feature map to realize efficient and accurate target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing and point cloud data processing, and specifically to a method, system, device and medium for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion. Background Art

[0002] 3D object detection is an important task in computer vision, which aims to detect and locate objects in a 3D scene. Different from traditional 2D object detection, 3D object detection not only focuses on the position of objects on the image plane, but also requires estimating the position, pose and size of objects in 3D space. This makes 3D object detection more widely used in many applications, such as autonomous driving, robot navigation, augmented reality, etc.

[0003] Single-mode 3D target detection refers to the use of one sensor information, such as only using LiDAR or only using a camera. The limitation of a single sensor may lead to insufficient environmental information, making it difficult to accurately capture 3D information in certain scenes or conditions. In addition, single-mode 3D target detection is more dependent on the characteristics of the environment and is more sensitive to factors such as ambient lighting, shadows, and occlusions, which limits its robustness in complex and changing environments. Multimodal 3D target detection refers to the use of information from different sensors, fully combining the data of multiple sensors such as LiDAR and cameras, to fully and accurately understand the 3D scene and improve the accuracy of 3D target detection. Among them, the camera can provide rich object appearance features, which can better complete target detection and classification, and the point cloud can provide accurate geometric features and spatial information of the object, which can better complete the positioning of the target in 3D space. However, the existing multimodal 3D target detection technology often focuses on the data fusion level, and does not make full use of the feature data of a single mode in the fusion process, resulting in a certain degree of information waste, which in turn leads to the existing methods. The algorithm is relatively complex at the data fusion level, and the data processing efficiency and real-time performance are poor. Summary of the invention

[0004] In view of the problems in the prior art of insufficient mining of single modal information, low utilization rate, and relatively complex algorithms at the data fusion level, the present invention proposes a method for three-dimensional target detection based on multimodal data of image depth estimation and point cloud fusion, as well as a system, device and medium for applying the method. The detection method obtains pixel-by-pixel depth data and foreground pixel data by performing monocular depth estimation and semantic segmentation on the initial image data of the target; then the pixel-by-pixel depth data is restored to a pseudo point cloud, and the pseudo point cloud is classified in combination with the foreground pixels to obtain pseudo point cloud classification information; then the initial point cloud data of the target is obtained, and fused with the pseudo point cloud data to obtain fused point cloud data; then the fused point cloud data is subjected to semantic refinement processing, voxel segmentation processing, and feature aggregation processing to obtain fused voxel high-dimensional features; then the fused voxel high-dimensional features are subjected to feature convolution operations to obtain a two-dimensional feature map; finally, the two-dimensional feature map is subjected to sliding window and anchor frame placement operations to achieve efficient and accurate target detection.

[0005] The technical solution of the present invention is:

[0006] The method for detecting three-dimensional objects using multimodal data based on image depth estimation and point cloud fusion comprises the following steps:

[0007] Step 1: Acquire initial image data of the target; perform monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data, and perform semantic segmentation on the initial image data to obtain foreground pixel data;

[0008] Step 2: using the pixel-by-pixel depth data obtained in step 1, convert each pixel in the initial image from a plane coordinate to a three-dimensional coordinate to obtain pseudo point cloud data of all pixels; and classifying the pseudo point cloud data according to the foreground pixel data obtained in step 1 to obtain pseudo point cloud classification information, and attaching the category information to the pseudo point cloud data as an additional feature;

[0009] Step 3: Obtain the initial point cloud data of the target, perform information statistics on the pseudo point cloud data and the initial point cloud data, and select all the initial point cloud data of the target within the range of 0-20 meters, and select 10% of the pseudo point cloud data in the farthest point sampling mode, and select all the pseudo point cloud data within a range greater than 20 meters, and select 10% of the initial point cloud data of the target in the farthest point sampling mode to obtain fused point cloud data;

[0010] Step 4: semantically refine the fused point cloud data obtained in step 3: locate the point cloud data carrying category information in the fused point cloud, use the set distance k as the radius, assign the category information to the neighboring points within the radius, and then perform columnar voxel division on the fused point cloud data to obtain fused voxels, and perform feature aggregation to obtain high-dimensional features of the fused voxels;

[0011] Step 5: performing a feature convolution operation on the fused voxel high-dimensional features obtained in step 4 to obtain a two-dimensional feature map;

[0012] Step 6: Perform sliding window and anchor frame placement operations on the two-dimensional feature map obtained in step 5 to obtain the target detection result.

[0013] Furthermore, in step 1, the process of performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data is as follows:

[0014] Performing an inverse projection matrix transformation operation on the initial image data to obtain the plane coordinates of all pixels in the initial image in the world coordinate system, and generating a set of depth information for each pixel to obtain a candidate set of pixel depth information;

[0015] The pixel depth information candidate set is normalized and linearly processed to obtain a depth feature vector set of all pixels;

[0016] The depth feature vector set is subjected to one-dimensional convolution, RELU activation function, and full connection layer to obtain the pixel-by-pixel depth data.

[0017] Furthermore, in step 1, the process of performing semantic segmentation on the initial image data to obtain foreground pixel data is as follows:

[0018] Normalizing and resizing the initial image data to obtain image data of appropriate size;

[0019] Performing two-dimensional convolution and maximum pooling operations on the image data of the appropriate size to obtain a high-level semantic feature map;

[0020] Processing the high-level semantic feature map by pixel classification probability to obtain a pixel-by-pixel category probability set;

[0021] Extracting the maximum value of the category probability from the pixel-by-pixel category probability set, and taking the classification result corresponding to the maximum value of the category probability as the pixel-by-pixel classification;

[0022] The obtained pixel-by-pixel classification information is subjected to category statistics to obtain pixels belonging to the foreground category as the foreground pixel data.

[0023] Furthermore, in step 2, the ray restoration operation is performed on each pixel to obtain the pseudo point cloud data of all pixels as follows:

[0024] According to the initial image data of the target, combined with the internal and external parameter information of the camera, the initial image pixels are converted to the camera coordinate system to obtain the plane coordinates of the pixels in the camera coordinate system;

[0025] Combined with the pixel-by-pixel depth data obtained in step 1, the plane coordinates of the pixels in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data.

[0026] Furthermore, in step 2, the pseudo point cloud data is classified to obtain the pseudo point cloud classification information as follows:

[0027] Create a blank tensor of the same size as the target's initial image and initialize all of them to 0;

[0028] According to the foreground pixel data obtained in step 1, after a position index operation, the value at the foreground pixel position corresponding to the blank tensor is modified to 1, thereby obtaining a foreground pixel mask;

[0029] According to the foreground pixel mask, a pseudo point cloud generated by foreground pixels is obtained, which is marked as foreground pseudo point cloud, and the rest are marked as background pseudo point cloud, and the classification information of the pseudo point cloud is obtained, and the classification information is embedded into the pseudo point cloud; specifically: the category information is attached to the information field, and the original information of each point has three-dimensional coordinates and reflectivity. After embedding the category, each point has three-dimensional coordinates, reflectivity, and category information.

[0030] Furthermore, in step 3, the process of performing distance division and fusion operation on the pseudo point cloud data and the initial point cloud data is as follows:

[0031] According to the range of the initial point cloud data, a step unit is selected and set to divide the pseudo point cloud data and the initial point cloud data by distance, so as to obtain a point cloud distance classification set;

[0032] The pseudo point cloud data and the initial point cloud data are sampled according to the sampling quantity within the predefined set step unit and the point cloud distance classification set, and then the sampled pseudo point cloud and the initial point cloud are merged to obtain fused point cloud data.

[0033] Furthermore, in step 5, the process of performing feature convolution operation is:

[0034] According to the preset voxel size and the fused voxel high-dimensional features, the fused voxel high-dimensional features are mapped to a plane by reducing the number of channels to obtain a plane feature map;

[0035] The planar feature map is subjected to two-dimensional convolution and pooling operations to obtain a two-dimensional feature map.

[0036] Furthermore, the process of sliding the window and placing the anchor frame in step 6 is as follows:

[0037] A fixed-size window is used to slide on the two-dimensional feature map with a fixed step. At each window position, several anchor boxes of different proportions are placed as candidate regions. Feature extraction is performed on each candidate region, and the extracted features are input into the target classifier to determine whether the window contains the target object. Finally, the detection results in all windows are combined to obtain the final target detection result.

[0038] Based on the above method, the present invention also proposes a multimodal data three-dimensional target detection system, comprising:

[0039] A pixel depth data generation module is used to perform monocular depth estimation on the acquired target initial image data to obtain pixel-by-pixel depth information;

[0040] A foreground pixel generation module is used to perform semantic segmentation on the acquired target initial image data to obtain foreground pixel data;

[0041] A pseudo point cloud classification information generation module is used to perform a ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo point cloud data, and classify the pseudo point cloud data using the foreground pixel data output by the foreground pixel generation module to obtain pseudo point cloud classification information;

[0042] A fused point cloud data generating module is used to divide the pseudo point cloud data and the initial point cloud data by distance to obtain a point cloud distance classification set, and based on the point cloud distance classification set, fuse the pseudo point cloud data and the initial point cloud data within a set distance range to form the fused point cloud data;

[0043] A three-dimensional feature fusion module is used to perform semantic refinement processing on the fused point cloud data and perform voxel division to obtain voxel data corresponding to all point clouds. The voxel data is subjected to feature aggregation processing to obtain fused voxel high-dimensional features;

[0044] A two-dimensional feature map generation module is used to perform feature convolution coding on the fused voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map;

[0045] The detection result generation module is used to perform sliding window and anchor frame placement operations on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

[0046] In addition, the present invention also provides a computer device and a computer-readable storage medium:

[0047] The computer device includes a processor and a memory. When the processor executes the computer program stored in the memory, the multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion is implemented.

[0048] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion is implemented.

[0049] Beneficial Effects

[0050] The present invention provides a multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion. The method performs monocular depth estimation and semantic segmentation on the initial image data of the target to obtain pixel-by-pixel depth data and foreground pixel data; then, the pixel-by-pixel depth data is restored to a pseudo point cloud, and the pseudo point cloud is classified in combination with the foreground pixels to obtain pseudo point cloud classification information; then, the initial point cloud data of the target is obtained, and a fusion operation is performed with the pseudo point cloud data to obtain fused point cloud data; then, semantic refinement processing, voxel segmentation processing, and feature aggregation processing are performed on the fused point cloud data to obtain fused voxel high-dimensional features; then, feature convolution operations are performed on the fused voxel high-dimensional features to obtain a two-dimensional feature map; finally, a sliding window and anchor frame placement operations are performed on the two-dimensional feature map to achieve efficient and accurate target detection.

[0051] The present invention provides a multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion. The method utilizes an image monocular depth estimation method to fuse image pixels with real point cloud data, thereby effectively solving the problem of information waste in conventional multimodal three-dimensional target detection schemes.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] Figure 1 : Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0055] Embodiments of the present invention are described in detail below. The embodiments are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0056] This embodiment proposes a multimodal data 3D target detection method based on image depth estimation and point cloud fusion, comprising the following steps:

[0057] Step 1: Acquire initial image data of the target; perform monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data, and perform semantic segmentation on the initial image data to obtain foreground pixel data.

[0058] In this embodiment, a camera device is used to obtain an image within the range of the target to be detected to form initial image data of the target. In order to improve the training efficiency of the model, data enhancement processing is performed on the initial image data. The data enhancement operations include but are not limited to: random cropping and random flipping.

[0059] Then perform monocular depth estimation on the initial image data of the target area to obtain pixel-by-pixel depth data:

[0060] According to the camera internal and external parameter matrix, the initial image data is subjected to an inverse projection matrix transformation operation, and the pixel coordinates are converted to the camera coordinate system, and then to the world coordinate system, to obtain the plane coordinates of all pixels in the initial image in the world coordinate system; a set of depth information is generated for each pixel at a fixed interval for candidate, to obtain a candidate set of pixel depth information; the candidate set of pixel depth information is normalized, linearly processed, and subjected to maximum pooling operations to predict depth information, to obtain a set of depth feature vectors for all pixels, and the depth feature vector set is subjected to one-dimensional convolution, RELU activation function, and fully connected layer, and the depth information with the highest score is retained to represent the pixel depth information, and the pixel-by-pixel depth data is obtained after nonlinear transformation. After this step is completed, the depth information corresponding to the image pixel is obtained.

[0061] Then, semantic segmentation is performed on the initial image data of the target area to obtain the foreground pixel data:

[0062] The initial image data is normalized and rescaled to obtain image data of a suitable size (size is 370×1240); the image data of the suitable size is subjected to two-dimensional convolution and maximum pooling operations to obtain a high-level semantic feature map; the high-level semantic feature map is processed by pixel classification probability to obtain a pixel-by-pixel category probability set; the maximum category probability is extracted from the pixel-by-pixel category probability set, and the classification result corresponding to the maximum category probability is used as the pixel-by-pixel classification; the obtained pixel-by-pixel classification information is subjected to category statistics to obtain pixels belonging to the foreground category as the foreground pixel data.

[0063] Step 2: For the pixel-by-pixel depth data obtained in step 1, each pixel in the initial image is converted from plane coordinates to three-dimensional coordinates to obtain pseudo point cloud data of all pixels; and based on the foreground pixel data obtained in step 1, the pseudo point cloud data is classified to obtain pseudo point cloud classification information.

[0064] First, according to the initial image data of the target and combined with the internal and external parameter information of the camera, the initial image pixels are converted to the camera coordinate system to obtain the plane coordinates of the pixels in the camera coordinate system; then, combined with the pixel-by-pixel depth data obtained in step 1, the plane coordinates of the pixels in the camera coordinate system are converted to three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data.

[0065] In order to distinguish the point cloud categories (such as vehicles and non-vehicles) in the pseudo point cloud data, the pseudo point cloud is classified according to the pseudo point cloud coordinate information and foreground pixel data:

[0066] Create a blank tensor of the same size as the target's initial image and initialize all of them to 0;

[0067] According to the foreground pixel data obtained in step 1, the coordinate position thereof in the initial image is obtained, and a unique index is assigned to each foreground pixel;

[0068] According to the position index, the value at the corresponding foreground pixel position in the blank tensor is modified to 1 to obtain the foreground pixel mask;

[0069] According to the foreground pixel mask, the position index is performed in the pseudo point cloud data space to obtain the pseudo point cloud generated by the foreground (vehicle) pixels, which is marked as the foreground pseudo point cloud, and the rest is marked as the background pseudo point cloud to obtain the pseudo point cloud classification information.

[0070] Step 3: Obtain the initial point cloud data of the target, perform information statistics on the pseudo point cloud data and the initial point cloud data, select all the initial point clouds of the target and select 10% of the pseudo point cloud data by farthest point sampling within the range of 0-20 meters, and select all the pseudo point cloud data and select 10% of the initial point cloud data of the target by farthest point sampling within the range greater than 20 meters to obtain fused point cloud data.

[0071] Use laser radar to obtain point cloud data within the range of the target to be detected as the initial point cloud data of the target area. The point cloud data includes but is not limited to three-dimensional coordinate information and reflection intensity information.

[0072] According to the initial point cloud data, the shortest distance and the farthest distance in the point cloud are determined, and information statistics are performed on the pseudo point cloud data and the initial point cloud data to obtain a plurality of distance division intervals. In each distance division interval, point clouds included in the distance range are collected to form a point cloud distance classification set;

[0073] The pseudo point cloud data and the initial point cloud data are sampled according to the sampling quantity within the predefined set step unit and the point cloud distance classification set, and then the sampled pseudo point cloud and the initial point cloud are merged to obtain fused point cloud data.

[0074] Step 4: semantically refine the fused point cloud data obtained in step 3, locate the point cloud carrying category information in the fused point cloud, take k as the radius, assign the category information to the neighboring points within the radius, and then perform columnar voxel division on the fused point cloud data to obtain fused voxels, and perform feature aggregation to obtain high-dimensional features of the fused voxels;

[0075] In step 2, the classification information has been embedded into the pseudo point cloud. First, it is necessary to locate the point cloud carrying the classification information in the fused point cloud data.

[0076] With the point cloud carrying the category information as the center point (which can be called the information center point), and k as the radius, we get the set of neighboring points within the radius area, capture the point cloud data closest to the information center point, and get the nearest neighbor point set of the information center point cloud. The classification information of the information center point is assigned to its corresponding point cloud nearest neighbor point, and the association between the foreground pseudo point cloud and the actual initial point cloud is established, so that each point in the fused point cloud carries the classification information from its nearest neighbor pseudo point cloud.

[0077] Afterwards, the fused point cloud data is processed by columnar voxel division, and the point cloud information is mapped to a regular three-dimensional voxel grid to obtain fused voxels; high-dimensional feature extraction is performed on the fused voxels, and the feature information of the point cloud in the voxels is integrated to obtain high-dimensional features of the fused voxels.

[0078] Step 5: Perform feature convolution operation on the fused voxel high-dimensional features obtained in step 4 to obtain a two-dimensional feature map.

[0079] According to the preset voxel size and the fused voxel high-dimensional features, the fused voxel high-dimensional features are mapped to a plane by reducing the number of channels to obtain a plane feature map, and the plane feature map is subjected to two-dimensional convolution and pooling operations to obtain a two-dimensional feature map. Specifically:

[0080] The center of the fused voxel is projected onto the horizontal plane to form a top view; the top view is divided into grids of fixed size, each grid corresponding to an area in the top view; the number of voxels in each grid is counted, and the average operation is performed to obtain a blank plane feature map;

[0081] Using the space-to-channel approach, the voxel high-dimensional features are filled into the blank plane feature map to obtain the plane feature map;

[0082] The planar feature map is subjected to two-dimensional convolution, maximum pooling, and nonlinear transformation to obtain a two-dimensional feature map.

[0083] Step 6: Perform sliding window and anchor frame placement operations on the two-dimensional feature map obtained in step 5 to obtain the target detection result.

[0084] Use a fixed-size window to slide on the two-dimensional feature map with a fixed step to capture information at different positions in the map; place several anchor boxes of different proportions at each window position as candidate regions; perform feature extraction on each candidate region, that is, intercept the corresponding feature map area in each anchor frame, and obtain the feature representation related to the area through convolutional neural networks or other feature extraction methods.

[0085] The extracted features are input into the target classifier to determine whether the window contains the target object; the pre-trained target classifier is usually a deep learning model, which can be a convolutional neural network (CNN) or other models suitable for classification tasks. The classifier evaluates each candidate area and outputs a probability or confidence score indicating whether the area contains the target object.

[0086] Finally, the detection results in all windows are combined to obtain the final target detection result.

[0087] Based on the above method, this embodiment also proposes a multimodal data three-dimensional object detection system, including:

[0088] A pixel depth data generation module is used to perform monocular depth estimation on the acquired target initial image data to obtain pixel-by-pixel depth information;

[0089] A foreground pixel generation module is used to perform semantic segmentation on the acquired target initial image data to obtain foreground pixel data;

[0090] A pseudo point cloud classification information generation module is used to perform a ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo point cloud data, and classify the pseudo point cloud data using the foreground pixel data output by the foreground pixel generation module to obtain pseudo point cloud classification information;

[0091] A fused point cloud data generating module is used to divide the pseudo point cloud data and the initial point cloud data by distance to obtain a point cloud distance classification set, and based on the point cloud distance classification set, fuse the pseudo point cloud data and the initial point cloud data within a set distance range to form the fused point cloud data;

[0092] A three-dimensional feature fusion module is used to perform semantic refinement processing on the fused point cloud data and perform voxel division to obtain voxel data corresponding to all point clouds. The voxel data is subjected to feature aggregation processing to obtain fused voxel high-dimensional features;

[0093] A two-dimensional feature map generation module is used to perform feature convolution coding on the fused voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map;

[0094] The detection result generation module is used to perform sliding window and anchor frame placement operations on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

[0095] In addition, the present invention also provides a computer device and a computer-readable storage medium:

[0096] The computer device includes a processor and a memory. When the processor executes the computer program stored in the memory, the multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion is implemented.

[0097] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion is implemented.

[0098] The present embodiment provides a multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion, which performs monocular depth estimation and semantic segmentation on the initial image data of the target to obtain pixel-by-pixel depth data and foreground pixel data; then, the pixel-by-pixel depth data is restored to a pseudo point cloud, and the pseudo point cloud is classified in combination with the foreground pixels to obtain pseudo point cloud classification information; then, the initial point cloud data of the target is obtained, and a fusion operation is performed with the pseudo point cloud data to obtain fused point cloud data; then, the fused point cloud data is subjected to semantic refinement processing, voxel segmentation processing, and feature aggregation processing to obtain fused voxel high-dimensional features; then, feature convolution operations are performed on the fused voxel high-dimensional features to obtain a two-dimensional feature map; finally, a sliding window and anchor frame placement operations are performed on the two-dimensional feature map to achieve efficient and accurate target detection.

[0099] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion, characterized in that: The following steps are involved: Step 1: Obtain the initial image data of the target; Performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data, and performing semantic segmentation on the initial image data to obtain foreground pixel data; Step 2: Using the pixel-by-pixel depth data obtained in step 1, each pixel in the initial image is converted from a plane coordinate to a three-dimensional coordinate to obtain pseudo point cloud data of all pixels; And according to the foreground pixel data obtained in step 1, the pseudo point cloud data is classified to obtain pseudo point cloud classification information, and the category information is attached to the pseudo point cloud data as an additional feature; Step 3: Obtain the initial point cloud data of the target, perform information statistics on the pseudo point cloud data and the initial point cloud data, and select all the initial point cloud data of the target within the range of 0-20 meters, and select 10% of the pseudo point cloud data in the farthest point sampling mode, and select all the pseudo point cloud data within a range greater than 20 meters, and select 10% of the initial point cloud data of the target in the farthest point sampling mode to obtain fused point cloud data; Step 4: semantically refine the fused point cloud data obtained in step 3: locate the point cloud data carrying category information in the fused point cloud, use the set distance k as the radius, assign the category information to the neighboring points within the radius, and then perform columnar voxel division on the fused point cloud data to obtain fused voxels, and perform feature aggregation to obtain high-dimensional features of the fused voxels; Step 5: performing a feature convolution operation on the fused voxel high-dimensional features obtained in step 4 to obtain a two-dimensional feature map; Step 6: Perform sliding window and anchor frame placement operations on the two-dimensional feature map obtained in step 5 to obtain the target detection result.

2. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: In step 1, the process of performing monocular depth estimation on the initial image data to obtain pixel-by-pixel depth data is as follows: Performing an inverse projection matrix transformation operation on the initial image data to obtain the plane coordinates of all pixels in the initial image in the world coordinate system, and generating a set of depth information for each pixel to obtain a candidate set of pixel depth information; The pixel depth information candidate set is normalized and linearly processed to obtain a depth feature vector set of all pixels; The depth feature vector set is subjected to one-dimensional convolution, RELU activation function, and full connection layer to obtain the pixel-by-pixel depth data.

3. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: In step 1, the process of performing semantic segmentation on the initial image data to obtain foreground pixel data is as follows: Normalizing and resizing the initial image data to obtain image data of appropriate size; Performing two-dimensional convolution and maximum pooling operations on the image data of the appropriate size to obtain a high-level semantic feature map; Processing the high-level semantic feature map by pixel classification probability to obtain a pixel-by-pixel category probability set; Extracting the maximum value of the category probability from the pixel-by-pixel category probability set, and taking the classification result corresponding to the maximum value of the category probability as the pixel-by-pixel classification; The obtained pixel-by-pixel classification information is subjected to category statistics to obtain pixels belonging to the foreground category as the foreground pixel data.

4. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: In step 2, the ray restoration operation is performed on each pixel to obtain the pseudo point cloud data of all pixels as follows: According to the initial image data of the target, combined with the internal and external parameter information of the camera, the initial image pixels are converted to the camera coordinate system to obtain the plane coordinates of the pixels in the camera coordinate system; Combined with the pixel-by-pixel depth data obtained in step 1, the plane coordinates of the pixels in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain the pseudo point cloud data. Furthermore, in step 2, the pseudo point cloud data is classified to obtain the pseudo point cloud classification information as follows: Create a blank tensor of the same size as the target's initial image and initialize all of them to 0; According to the foreground pixel data obtained in step 1, after a position index operation, the value at the foreground pixel position corresponding to the blank tensor is modified to 1, thereby obtaining a foreground pixel mask; According to the foreground pixel mask, a pseudo point cloud generated by the foreground pixels is obtained, which is marked as a foreground pseudo point cloud, and the rest are marked as background pseudo point clouds, and classification information of the pseudo point cloud is obtained, and the classification information is embedded into the pseudo point cloud; Specifically, the category information is attached to the information field. The original information of each point has three-dimensional coordinates and reflectivity. After embedding the category, each point has three-dimensional coordinates, reflectivity, and category information.

5. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: In step 3, the process of performing distance division and fusion operation on the pseudo point cloud data and the initial point cloud data is as follows: According to the range of the initial point cloud data, a step unit is selected and set to divide the pseudo point cloud data and the initial point cloud data by distance, so as to obtain a point cloud distance classification set; The pseudo point cloud data and the initial point cloud data are sampled according to the sampling quantity within the predefined set step unit and the point cloud distance classification set, and then the sampled pseudo point cloud and the initial point cloud are merged to obtain fused point cloud data.

6. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: In step 5, the process of performing feature convolution operation is: According to the preset voxel size and the fused voxel high-dimensional features, the fused voxel high-dimensional features are mapped to a plane by reducing the number of channels to obtain a plane feature map; The planar feature map is subjected to two-dimensional convolution and pooling operations to obtain a two-dimensional feature map.

7. The method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to claim 1, characterized in that: The process of sliding the window and placing the anchor frame in step 6 is as follows: A fixed-size window is used to slide on the two-dimensional feature map with a fixed step. At each window position, several anchor boxes of different proportions are placed as candidate regions. Feature extraction is performed on each candidate region, and the extracted features are input into the target classifier to determine whether the window contains the target object. Finally, the detection results in all windows are combined to obtain the final target detection result.

8. A multimodal data three-dimensional target detection system, characterized in that: include A pixel depth data generation module is used to perform monocular depth estimation on the acquired target initial image data to obtain pixel-by-pixel depth information; A foreground pixel generation module is used to perform semantic segmentation on the acquired target initial image data to obtain foreground pixel data; A pseudo point cloud classification information generation module is used to perform a ray restoration operation on the pixel-by-pixel depth information output by the pixel depth data generation module to obtain pseudo point cloud data, and classify the pseudo point cloud data using the foreground pixel data output by the foreground pixel generation module to obtain pseudo point cloud classification information; A fused point cloud data generating module is used to divide the pseudo point cloud data and the initial point cloud data by distance to obtain a point cloud distance classification set, and based on the point cloud distance classification set, fuse the pseudo point cloud data and the initial point cloud data within a set distance range to form the fused point cloud data; A three-dimensional feature fusion module is used to perform semantic refinement processing on the fused point cloud data and perform voxel division to obtain voxel data corresponding to all point clouds. The voxel data is subjected to feature aggregation processing to obtain fused voxel high-dimensional features; A two-dimensional feature map generation module is used to perform feature convolution coding on the fused voxel high-dimensional features output by the three-dimensional feature fusion module to obtain a two-dimensional feature map; The detection result generation module is used to perform sliding window and anchor frame placement operations on the two-dimensional feature map output by the two-dimensional feature map generation module to generate a target detection result.

9. A computer device, characterized in that: The computer device includes a processor and a memory, and when the processor executes the computer program stored in the memory, it implements the multimodal data three-dimensional target detection method based on image depth estimation and point cloud fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for three-dimensional target detection based on multimodal data based on image depth estimation and point cloud fusion according to any one of claims 1 to 7 is implemented.

Images