Camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization, which comprises the following steps: acquiring binocular image data and laser radar point cloud data; based on the binocular image data, binocular image features are extracted, a stereo matching network is used for estimation and prediction, and camera parameters are mapped to a three-dimensional space to generate pseudo point cloud data; the point cloud data and the pseudo point cloud data are respectively subjected to feature coding to obtain point cloud feature information and pseudo point cloud feature information, the point cloud feature information is used for supervising training of a pseudo point cloud feature decoder, and optimized pseudo point cloud data is reconstructed and generated; in combination with the binocular image features, a multi-modal feature fusion network is used for cross-modal fusion, a detection head is used to construct a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization; and the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is used for target detection on the binocular image to generate a target detection result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization. BACKGROUND

[0002] The vehicle-mounted sensor technology of the automatic driving system mainly depends on two core components of lidar and camera. In the research and application of target detection algorithms, the algorithms are mainly divided into three categories: lidar point cloud-based algorithms, camera image-based algorithms, and multi-modal fusion-based algorithms. Among them, the technical solutions based on lidar point cloud and multi-modal fusion can accurately identify and locate targets through prior information, while the pure vision three-dimensional target detection model is difficult to reconstruct the three-dimensional scene due to the lack of real depth information of image data, resulting in a significant difference in performance from the former two.

[0003] At present, in order to narrow the gap between pure vision three-dimensional target detection and point cloud and multi-modal fusion solutions, an algorithm based on pseudo lidar is proposed. The algorithm generates pseudo point cloud data through the inverse projection of depth images, and performs target detection with the help of mature point cloud processing algorithms, thereby improving the performance of pure vision three-dimensional target detection and making it closer to the point cloud and multi-modal fusion-based solutions. Although this algorithm has advantages in cost and data acquisition scheme, there are still challenges in depth estimation accuracy, stability in complex environments, data intensity, and real-time performance and performance of the algorithm. For example, the accuracy limitation of the depth estimation algorithm will cause the generated pseudo point cloud to have large errors in target edge distribution and local structure information inside the template, and the density of the pseudo point cloud may increase the computational burden of the algorithm, affecting the detection speed and accuracy. SUMMARY

[0004] The application provides a camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization. By finely adjusting and optimizing the pseudo point cloud data, introducing the point cloud feature information of the lidar point cloud data for feature-level supervision, and reconstructing the feature distribution of the optimized pseudo point cloud, the problem of large errors in target edge distribution and local structure information inside the target of the generated pseudo point cloud data can be solved. At the same time, the multi-modal feature fusion network is used to perform cross-modal fusion on the features of the pseudo point cloud data and the features of the binocular image, so as to fully utilize the rich color and texture information of the image data and the three-dimensional spatial structure information of the pseudo point cloud, realize the complementary advantages of the two at the feature level, and further significantly improve the accuracy and robustness of the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization.

[0005] A camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization, comprising the following steps:

[0006] acquire binocular image data of a vehicle-mounted camera and point cloud data of a laser radar;

[0007] extract binocular image features based on the binocular image data, and estimate and predict by using a stereo matching network to generate pseudo point cloud data in a three-dimensional space in combination with camera parameters;

[0008] respectively encode the point cloud data and the pseudo point cloud data to obtain point cloud feature information and pseudo point cloud feature information, and supervise training of a pseudo point cloud feature decoder by using the point cloud feature information to reconstruct and generate optimized pseudo point cloud data;

[0009] based on the optimized pseudo point cloud data, in combination with the binocular image features, perform cross-modal fusion by using a multi-modal feature fusion network, and construct a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization by using a detection head;

[0010] perform target detection on the binocular image by using the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization to generate a target detection result.

[0011] By finely adjusting and optimizing the pseudo point cloud data, introducing point cloud feature information of the laser radar point cloud data for feature-level supervision, reconstructing the feature distribution of the optimized pseudo point cloud, the problem of large error in target edge distribution and local structure information within the target of the generated pseudo point cloud data can be solved; at the same time, the multi-modal feature fusion network is used to perform cross-modal fusion on the features of the pseudo point cloud data and the binocular image features, so that the rich color and texture information of the image data and the three-dimensional spatial structure information of the pseudo point cloud are fully utilized, the advantages of the two at the feature level are complementary, and the accuracy and robustness of the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization are significantly improved.

[0012] Further, based on the binocular image data, the binocular image features are extracted, and the stereo matching network is used for estimation and prediction, and the process of generating the pseudo point cloud data in combination with mapping into a three-dimensional space includes the following steps:

[0013] based on the binocular image data, an image feature generation network is used to obtain binocular image features;

[0014] based on the binocular image features, left and right image features are connected to form a 4D depth cost volume, a stereo matching network is used to evaluate the credibility of each real depth value, and a prediction cost is generated;

[0015] convert the prediction cost into the probability of each real depth value, combine the weighted sum of all real depth values and their corresponding probabilities, calculate the predicted depth value, and use a smoothing loss function to supervise the difference between the real depth value and the predicted depth value;

[0016] The pixel points and the corresponding predicted depth values are mapped into a three-dimensional space in combination with camera parameters to generate pseudo point cloud data;

[0017] An expression for generating the pseudo point cloud data is:

[0018] ;

[0019] ;

[0020] In the formula, represents the pseudo point cloud data; represents a position on an X axis in the three-dimensional space; represents a position on a Y axis in the three-dimensional space; represents a position on a Z axis in the three-dimensional space; represents a physical distance between any two pixels on the X axis; represents a physical distance between any two pixels on the Y axis; represents a focal length of the camera; represents a position of an origin on the X axis in a pixel coordinate system; represents a position of the origin on the Y axis in the pixel coordinate system; represents a depth value corresponding to each pixel point in the pixel coordinate system, represents a coordinate of the corresponding pixel point on the X axis, represents a coordinate of the corresponding pixel point on the Y axis.

[0021] Further, based on the binocular image data, the process of obtaining binocular image features by using an image feature generation network includes the following steps:

[0022] Based on the binocular image data, the original image features output by each level of residual module are obtained through convolution layers and multiple residual modules;

[0023] The image features output by the last residual module are compressed by using an adaptive average pooling layer, and are spliced with the original image features output by the selected residual modules to generate binocular image features through convolution;

[0024] An expression for generating the binocular image features is:

[0025] ;

[0026] In the formula, represents the binocular image features; represents an image feature generation network model; represents the binocular image data.

[0027] Further, the point cloud data and the pseudo point cloud data are respectively feature encoded to obtain point cloud feature information and pseudo point cloud feature information, and the point cloud feature information is used to supervise training of a pseudo point cloud feature decoder to reconstruct and generate optimized pseudo point cloud data.

[0028] Based on the point cloud data, a point cloud feature encoder is used for encoding to generate point cloud feature information;

[0029] Based on the pseudo point cloud data, a pseudo point cloud feature encoder is used for encoding to generate pseudo point cloud feature information;

[0030] The point cloud feature information is used as a supervision signal to train the pseudo point cloud data through a pseudo point cloud feature decoder, total training loss is calculated and parameters of the pseudo point cloud encoder and the pseudo point cloud decoder are updated;

[0031] The updated pseudo point cloud feature encoder and the pseudo point cloud feature decoder are cascaded to reconstruct data and optimize spatial distribution of the pseudo point cloud feature information to generate optimized pseudo point cloud data.

[0032] Further, the point cloud encoder and the pseudo point cloud encoder have the same structure and share weights, and the encoding process specifically includes the following steps:

[0033] Based on the point cloud data or the pseudo point cloud data, a SAB module in a PointNet++ backbone network is used to extract multi-scale point cloud features to generate weighted features of the point cloud data or the pseudo point cloud data;

[0034] Based on the weighted features, a Transformer network architecture is used for self-attention reorganization to generate point cloud feature information or pseudo point cloud feature information;

[0035] The expression of the self-attention reorganization is:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] In the formula, represents the dimension of the feature vector; represents the input feature vector; represents the weighted feature obtained after the self-attention mechanism; represents the output feature after enhancement; Indicates the layer normalization function; Indicates the output function; Represents a multilayer perceptron; Indicates a query; Indicates key; Represents a value matrix; Indicates the weight of the query; Indicates the weight of the key; A matrix that represents a value matrix.

[0042] Furthermore, during the training process of the pseudo-point cloud feature decoder, the training loss includes the supervision loss of the intermediate layer features and the reconstruction loss of the reconstructed pseudo-point cloud data.

[0043] The supervision loss for intermediate layer features is calculated using Euclidean distance, and its expression is as follows:

[0044] ;

[0045] In the formula, This represents the loss of the intermediate layer of the pseudo-point cloud feature decoder;

[0046] The reconstruction loss for reconstructing pseudo-point cloud data is calculated using the chamfer distance, and its expression is as follows:

[0047] ;

[0048] In the formula, Indicates the loss based on the reconstructed value; Indicates the total number of point clouds;

[0049] The expression for the total training loss is:

[0050] ;

[0051] In the formula, This represents the total loss during the training process; , These represent the weights of the intermediate layer loss and the reconstructed loss weights, respectively.

[0052] Furthermore, based on optimized pseudo-point cloud data and combined with binocular image features, a multimodal feature fusion network is used for cross-modal fusion. The process of constructing a camera multimodal fusion 3D target detection model based on pseudo-point cloud feature optimization using the detection head specifically includes the following steps:

[0053] Based on optimized pseudo-point cloud data, a feature generation network is used to extract voxel features from disordered optimized pseudo-point cloud data, which are then used as optimized pseudo-point cloud features.

[0054] The expression for pseudo-point cloud features is:

[0055] ;

[0056] In the formula, represents the optimized pseudo point cloud feature information; represents the optimized pseudo point cloud data; represents a feature generation network;

[0057] The optimized pseudo point cloud feature and the binocular image feature are respectively input into a BEV feature generation network to form a BEV feature map of the pseudo point cloud mode and a BEV feature map of the binocular image;

[0058] The BEV feature map of the pseudo point cloud mode and the BEV feature map of the binocular image are input into a multi-modal feature fusion network for initial fusion splicing, and a convolutional neural network is used for secondary fusion of the spliced features to obtain a fusion feature;

[0059] The expression of the fusion feature is:

[0060] ;

[0061] In the formula, represents the fusion feature of the pseudo point cloud feature and the binocular image feature; represents a multi-modal feature fusion network; represents the BEV feature map of the pseudo point cloud mode; represents the BEV feature map of the binocular image;

[0062] The fusion feature is taken as an input of a detection head, and a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is trained and constructed.

[0063] A system of a camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization, comprising:

[0064] A data acquisition module is configured to acquire binocular image data of a vehicle-mounted camera and point cloud data of a laser radar;

[0065] A data generation module is configured to extract binocular image features based on the binocular image data, and calculate a depth map in combination with camera parameters to generate pseudo point cloud data mapped to a three-dimensional space;

[0066] A data optimization module is configured to respectively perform feature encoding on the point cloud data and the pseudo point cloud data, to obtain point cloud feature information and pseudo point cloud feature information, and to supervise training of a pseudo point cloud feature decoder using the point cloud feature information to reconstruct and generate optimized pseudo point cloud data;

[0067] A model construction module is configured to perform cross-modal fusion of the pseudo point cloud data based on optimization and binocular image features by using a multi-modal feature fusion network, and construct a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization by using a detection head.

[0068] A result generation module is configured to perform target detection on the binocular image by using the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization, and generate a target detection result.

[0069] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method as described above when executing the computer program.

[0070] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described above.

[0071] The present application has the following beneficial effects:

[0072] The present application can accurately correct the errors in the pseudo point cloud data by fine adjustment and optimization of the pseudo point cloud data, and can solve the problem of large errors in target edge distribution and local structure information in the generated pseudo point cloud data by introducing point cloud feature information of laser radar point cloud data for feature level supervision and reconstructing the feature distribution of the optimized pseudo point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 The flowchart of the present application;

[0074] Figure 2 The method flowchart of the present application;

[0075] Figure 3 The structural schematic diagram of the system in the present application;

[0076] Figure 4 The structural schematic diagram of the computer device in the present application. DETAILED DESCRIPTION

[0077] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0078] It is to be appreciated that various aspects of the embodiments within the scope of the appended claims are described below. It is to be understood that the aspects described herein can be implemented in a wide variety of forms and that any particular structure and / or function described herein is merely illustrative. Based on the disclosure provided, one skilled in the art should appreciate that an aspect described herein can be implemented, independently of any other aspects of the disclosure, and that an aspect can be combined with, or modified to include, features of any other aspect described herein. For example, a device can be implemented using any number of the aspects described herein. In addition, this device and / or method can be implemented using other structure and / or functionality not expressly described herein.

[0079] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. For one of ordinary skill in the art, the specific meaning of the above-mentioned terms in the present application can be understood in specific circumstances.

[0080] Embodiment 1

[0081] Figure 1 A camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization is shown. By fine adjustment and optimization of pseudo point cloud data, point cloud feature information of laser radar point cloud data is introduced for feature level supervision, and the feature distribution of the reconstructed and optimized pseudo point cloud is optimized. The problem of large error of target edge distribution and local structure information in the generated pseudo point cloud data can be solved. At the same time, a multi-modal feature fusion network is used to perform cross-modal fusion on the features of the pseudo point cloud data and the features of the binocular image, so as to fully utilize the rich color and texture information of the image data and the three-dimensional spatial structure information of the pseudo point cloud, realize the complementary advantages of the two at the feature level, and further significantly improve the accuracy and robustness of the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization. Specifically, the following steps are included:

[0082] S1: acquiring binocular image data of a vehicle-mounted camera and point cloud data of a laser radar;

[0083] S2: based on the binocular image data, extracting binocular image features, and using a stereo matching network to estimate and predict, and combining camera parameters to map to three-dimensional space to generate pseudo point cloud data;

[0084] S11: acquiring binocular image features based on binocular image data by using an image feature generation network;

[0085] wherein the expression for generating binocular image features is:

[0086] ;

[0087] In the formula, denotes binocular image features; denotes an image feature generation network model; denotes binocular image data.

[0088] Specifically, the process of acquiring binocular image features based on binocular image data by using an image feature generation network specifically includes the following steps:

[0089] S111: based on binocular image data, the original image features output by each level of residual module are obtained by processing through the convolution layer and multiple residual modules of the image feature generation network;

[0090] In this embodiment, the expression for generating the original image features output by each level of residual module is:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] In the formula, , , , , denote the original image features of each level obtained by sequentially passing through the convolution layer and each residual module, denotes a convolution function; denotes a residual function;

[0097] S112: compressing the image features output by the last residual module by using the adaptive average pooling layer in the SPP module, and splicing the original image features output by the selected remaining residual modules to generate binocular image features by convolution;

[0098] In this embodiment, the image features output by the last residual module are compressed by four scales by using the adaptive average pooling layer , and the expression is:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] wherein, , , , respectively represent image features compressed in dimension by layer after adaptive average pooling layer; represents an up-sampling function; represents an average pooling layer function;

[0104] In this embodiment, the features compressed features and original image features , original image features are merged, and the expression is:

[0105] ;

[0106] wherein, represents merged features; represents a feature merging function;

[0107] The expression of the binocular image feature generated by convolution is:

[0108] .

[0109] S12: based on the binocular image feature, the left and right image features are connected to form a 4D depth cost volume, and the credibility of each real depth value is evaluated by using a stereo matching network, and a prediction cost is generated;

[0110] wherein, the expression of the prediction cost is:

[0111] ;

[0112] wherein, represents a prediction cost; represents a stereo matching network structure; represents a 4D depth cost volume, and is expressed as , represents a depth channel, represents an image length, represents an image width, represents a feature channel;

[0113] S13: converting the prediction cost into a probability of each real depth value, combining a weighted sum of all real depth values and their corresponding probabilities, calculating a predicted depth value, and supervising the difference between the real depth value and the predicted depth value by using a smoothing loss function;

[0114] wherein the expression of the predicted depth value is:

[0115] ;

[0116] wherein, represents the predicted depth value; represents the real depth value; represents a normalization function; represents a maximum depth value;

[0117] wherein the expression of the predicted depth value supervised by the smoothing loss function is:

[0118] ;

[0119] wherein, represents the predicted depth value supervised by the smoothing loss function; represents the number of image pixel points, , represents the total number of image pixel points; represents the smoothing loss function.

[0120] S14: mapping the pixel points and their corresponding predicted depth values into a three-dimensional space by combining the camera parameters to generate pseudo point cloud data;

[0121] wherein the expression of the pseudo point cloud data is:

[0122] ;

[0123] ;

[0124] wherein, represents the pseudo point cloud data; represents a position on the X-axis in the three-dimensional space; represents a position on the Y-axis in the three-dimensional space; represents a position on the Z-axis in the three-dimensional space; represents a physical distance between any two pixels on the X-axis; represents a physical distance between any two pixels on the Y-axis; represents a focal length of the camera; represents a position of the origin on the X-axis in the pixel coordinate system; represents a position of the origin on the Y-axis in the pixel coordinate system; represents a depth value corresponding to each pixel point in a pixel coordinate system, represents a coordinate of the corresponding pixel point on the X axis, represents a coordinate of the corresponding pixel point on the Y axis.

[0125] S3: respectively encode the point cloud data and the pseudo point cloud data to obtain point cloud feature information and pseudo point cloud feature information, and supervise training of a pseudo point cloud feature decoder using the point cloud feature information to reconstruct and generate optimized pseudo point cloud data;

[0126] In the embodiment, the point cloud encoder and the pseudo point cloud encoder used for respectively encoding the point cloud data and the pseudo point cloud data have the same structure and share weights; the encoding process specifically includes the following steps:

[0127] Based on the point cloud data or the pseudo point cloud data, a SAB module in a PointNet++ backbone network is used to extract multi-scale point cloud features to generate weighted features of the point cloud data or the pseudo point cloud data;

[0128] Based on the weighted features, a Transformer network architecture is used for self-attention reorganization to generate point cloud feature information or pseudo point cloud feature information;

[0129] The expression of the self-attention reorganization is as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] In the formula, represents the dimension of the feature vector; represents the input feature vector; represents the weighted feature obtained after the self-attention mechanism; represents the output feature after enhancement; represents a layer normalization function; represents an output function; represents a multi-layer perceptron; represents a query; represents a key; represents a value matrix; represents the weight of the query; represents the weight of the key; represents the matrix of the value matrix.

[0136] In the embodiment, the pseudo point cloud feature decoder adopted is similar in structure to the pseudo point cloud encoder, and contains multiple stacked Transformer basic modules to control the degree of distribution adjustment of each level of the distribution optimization module.

[0137] Specifically, the point cloud data and the pseudo point cloud data are respectively encoded to obtain point cloud feature information and pseudo point cloud feature information, and the point cloud feature information is used to supervise the training of the pseudo point cloud feature decoder to reconstruct and generate optimized pseudo point cloud data, which specifically includes the following steps:

[0138] S31: Based on the point cloud data , a point cloud feature encoder is used for encoding to generate point cloud feature information, i.e., to obtain a set of point cloud feature vectors ;

[0139] S32: Based on the pseudo point cloud data , a pseudo point cloud feature encoder is used for encoding to generate pseudo point cloud feature information, i.e., to obtain a set of pseudo point cloud feature vectors ;

[0140] wherein, represents the number of levels of the feature network; represents the dimension of the feature vector of each level; represents the number of point sets sampled and obtained at each level.

[0141] S33: The point cloud feature information is used as a supervision signal to train the pseudo point cloud data through the pseudo point cloud feature decoder, to calculate a total training loss and update the parameters of the pseudo point cloud encoder and the pseudo point cloud decoder;

[0142] In the embodiment, during the training of the pseudo point cloud feature decoder, the training loss includes a supervision loss of the intermediate layer feature and a reconstruction loss of the reconstructed pseudo point cloud data.

[0143] wherein the supervision loss of the intermediate layer feature is calculated using the Euclidean distance, and the expression is:

[0144] ;

[0145] wherein, represents the loss of the intermediate layer of the pseudo point cloud feature decoder;

[0146] wherein the reconstruction loss of the reconstructed pseudo point cloud data is calculated using the chamfer distance, and the expression is:

[0147] ;

[0148] wherein, represents the loss based on the reconstruction; represents the total number of point clouds;

[0149] wherein the expression of the total loss of training is:

[0150] ;

[0151] In the formula, represents the total loss of the training process; , respectively represent the weight of the intermediate layer loss and the weight of the reconstructed loss.

[0152] S34: The updated pseudo point cloud feature encoder is cascaded with the pseudo point cloud feature decoder to perform data reconstruction and spatial distribution optimization on the pseudo point cloud feature information, and to generate optimized pseudo point cloud data .

[0153] S4: Based on the optimized pseudo point cloud data, combined with the binocular image features, a multi-modal feature fusion network is used for cross-modal fusion, and a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is constructed by using a detection head;

[0154] S41: Based on the optimized pseudo point cloud data , a feature generation network is used to extract unordered optimized pseudo point cloud data voxel features as the optimized pseudo point cloud features ;

[0155] wherein the expression of the pseudo point cloud features is:

[0156] ;

[0157] In the formula, represents the optimized pseudo point cloud feature information; represents the optimized pseudo point cloud data; represents the feature generation network;

[0158] wherein the extracted pseudo point cloud data voxel features , represent each direction in the three-dimensional space of the pseudo point cloud, and the specific values are determined by the range of the pseudo point cloud data and the size of the voxel, represents the dimension of the pseudo point cloud voxel features, which is in this embodiment.

[0159] S42: The optimized pseudo point cloud features and the binocular image features are input into the BEV feature generation network respectively to form the BEV feature map of the pseudo point cloud modality and the BEV feature map of the binocular image ;

[0160] S43: input the BEV feature map of the pseudo point cloud modality and the BEV feature map of the binocular image into a multi-modal feature fusion network for initial fusion splicing, and perform secondary fusion on the spliced features by using a convolutional neural network to obtain fusion features ;

[0161] The expression of the fusion features is as follows:

[0162] ;

[0163] In the formula, represents the fusion features of the pseudo point cloud features and the binocular image features; represents a multi-modal feature fusion network; represents the BEV feature map of the pseudo point cloud modality; represents the BEV feature map of the binocular image;

[0164] S44: input the fusion features as the input of a detection head, and train and construct a camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization.

[0165] S5: use the camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization to perform target detection on the binocular image to generate a target detection result.

[0166] Embodiment 2

[0167] As shown in Figure 2 , the embodiment provides a camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization, which specifically comprises the following steps:

[0168] T1: acquire binocular image data of a vehicle-mounted camera , and point cloud data of a laser radar ;

[0169] T2: based on the binocular image data , extract binocular image features , and use a stereo matching network to perform estimation and prediction, and combine camera parameters to map into a three-dimensional space to generate pseudo point cloud data ;

[0170] T11: based on the binocular image data , use an image feature generation network to acquire binocular image features ;

[0171] T12: based on the binocular image features ​​, connect left and right image features to form a 4D depth cost volume, and adopt a stereo matching network Evaluate the credibility of each real depth value and generate a prediction cost;

[0172] T13: Convert the prediction cost into a probability for each real depth value, combine the weighted sum of all real depth values and their corresponding probabilities, calculate the predicted depth value, and use a smoothing loss function to supervise the difference between the real depth value and the predicted depth value;

[0173] T14: Map the pixel point and its corresponding predicted depth value to three-dimensional space by combining the camera parameters to generate pseudo point cloud data ;

[0174] T3: Feature encoding is performed on point cloud data and pseudo point cloud data , respectively, to obtain point cloud feature information and pseudo point cloud feature information, and a pseudo point cloud feature decoder is trained using point cloud feature information to reconstruct and generate optimized pseudo point cloud data ;

[0175] T31: Based on point cloud data , a point cloud feature encoder is used for encoding to generate point cloud feature information, i.e., to obtain a set of point cloud feature vectors ;

[0176] T32: Based on pseudo point cloud data , a pseudo point cloud feature encoder is used for encoding to generate pseudo point cloud feature information, i.e., to obtain a set of pseudo point cloud feature vectors ;

[0177] T33: The point cloud feature information is used as a supervision signal to train the pseudo point cloud data through a pseudo point cloud feature decoder , calculate the total training loss and update the parameters of the pseudo point cloud encoder and the pseudo point cloud decoder ;

[0178] T34: The updated pseudo point cloud feature encoder and pseudo point cloud feature decoder are concatenated to reconstruct the data and optimize the spatial distribution of the pseudo point cloud feature information, generating optimized pseudo point cloud data .

[0179] T4: Based on the optimized pseudo point cloud data , combine the binocular image features , and use a multi-modal feature fusion network Cross-modal fusion is performed, and a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is constructed by using the detection head;

[0180] T41: based on the optimized pseudo point cloud data , a feature generation network is adopted to extract unordered optimized pseudo point cloud data voxel features as the optimized pseudo point cloud features ;

[0181] T42: the optimized pseudo point cloud features and binocular image features are input into a BEV feature generation network respectively to form a BEV feature map of the pseudo point cloud modality and a BEV feature map of the binocular image ;

[0182] T43: the BEV feature map of the pseudo point cloud modality and the BEV feature map of the binocular image are input into a multi-modal feature fusion network for initial fusion splicing, and a convolutional neural network is used for secondary fusion of the spliced features to obtain fusion features ;

[0183] T44: the fusion features are input into the detection head to train and construct a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization

[0184] T5: a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is used for target detection of the binocular image to generate a target detection result.

[0185] Embodiment 3

[0186] Based on the same design concept, as shown in Figure 3 , the embodiment provides a camera multi-modal fusion 3D target detection system based on pseudo point cloud feature optimization, which comprises a data acquisition module, a data generation module, a data optimization module, a model construction module, and a result generation module.

[0187] Specifically, the data acquisition module is used to acquire binocular image data of a vehicle-mounted camera and point cloud data of a laser radar.

[0188] Specifically, the data generation module is used to extract binocular image features based on the binocular image data, calculate a depth map in combination with camera parameters, and map the depth map to a three-dimensional space to generate pseudo point cloud data.

[0189] Specifically, the data optimization module is configured to encode features of the point cloud data and the pseudo point cloud data respectively, obtain point cloud feature information and pseudo point cloud feature information, and supervise training of a pseudo point cloud feature decoder using the point cloud feature information to reconstruct and generate optimized pseudo point cloud data.

[0190] Specifically, the model construction module is configured to perform cross-modal fusion using a multi-modal feature fusion network based on the optimized pseudo point cloud data and in combination with binocular image features, and construct a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization using a detection head.

[0191] Specifically, the result generation module is configured to perform target detection on binocular images using the camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization to generate a target detection result.

[0192] Embodiment 4

[0193] Based on the same technical concept, the embodiments of the present application also provide a computer device, which comprises a memory 1 and a processor 2, as shown in Figure 4 The memory 1 stores a computer program, and the processor 2 implements the method of any one of the above embodiments when executing the computer program.

[0194] The memory 1 comprises at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 1 can be an internal storage unit of the camera multi-modal fusion 3D target detection system based on pseudo point cloud feature optimization, such as a hard disk. In other embodiments, the memory 1 can also be an external storage device of the camera multi-modal fusion 3D target detection system based on pseudo point cloud feature optimization, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 1 can include both an internal storage unit and an external storage device of the camera multi-modal fusion 3D target detection system based on pseudo point cloud feature optimization. The memory 1 can be used not only to store application software and various data installed in the camera multi-modal fusion 3D target detection system based on pseudo point cloud feature optimization, such as codes of the camera multi-modal fusion 3D target detection system program based on pseudo point cloud feature optimization, but also to temporarily store data that has been output or will be output.

[0195] The processor 2 may, in some embodiments, be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes stored in the memory 1 or process data, such as executing a camera multi-modal fusion 3D target detection system program based on pseudo point cloud feature optimization.

[0196] The application further discloses a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in the above method embodiments are performed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0197] The application further discloses a computer program product for the application page content refreshing method, which comprises a computer readable storage medium storing program codes. The program codes comprise instructions for performing the steps of the method described in the above method embodiments. For details, refer to the above method embodiments, which will not be repeated here.

[0198] The application further discloses a computer program, which, when executed by a processor, implements any method of the above embodiments. The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0199] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the content not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

[0200] It should be noted that, in the description of the application, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance. In addition, in the description of the application, unless otherwise specified, "a plurality of" means at least two.

[0201] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specified logic functions or steps, and the various embodiments of the application include additional or fewer functions performed in the same order or in a different order, combined with one or more functions performed in the same or a different order, and the like, as will be appreciated by those skilled in the art. The various embodiments of the application each have a plurality of configurations and hardware implementations, as will be appreciated by those skilled in the art.

[0202] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, through software or firmware in storage media which are executable by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the above-described steps or methods can be implemented with any or a combination of the following technologies, which are all well-known in the art: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0203] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, and when the programs are executed, one or a combination of the steps of the method embodiment is included.

[0204] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0205] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0206] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A camera multi-modal fusion 3D target detection method based on pseudo point cloud feature optimization, characterized in that, The method comprises the following steps: Obtaining binocular image data of a vehicle-mounted camera and point cloud data of a laser radar; Based on the binocular image data, binocular image features are extracted, and a stereo matching network is used for estimation and prediction, and combined with camera parameters to map into a three-dimensional space to generate pseudo point cloud data; The point cloud data and the pseudo point cloud data are respectively encoded to obtain point cloud feature information and pseudo point cloud feature information, and the point cloud feature information is used to supervise the training of a pseudo point cloud feature decoder to reconstruct and generate optimized pseudo point cloud data, comprising the following steps: Based on the point cloud data, a point cloud feature encoder is used for encoding to generate point cloud feature information; Based on the pseudo point cloud data, a pseudo point cloud feature encoder is used for encoding to generate pseudo point cloud feature information; The point cloud feature information is used as a supervision signal to train the pseudo point cloud data through the pseudo point cloud feature decoder, the total training loss is calculated, and the parameters of the pseudo point cloud encoder and the pseudo point cloud decoder are updated; During the training of the pseudo point cloud feature decoder, the training loss comprises a supervision loss of intermediate layer features and a reconstruction loss of reconstructed pseudo point cloud data; The supervision loss of the intermediate layer features is calculated by using the Euclidean distance, and the expression is as follows: ； In the formula, denotes the loss of the pseudo point cloud feature decoder intermediate layer; The reconstruction loss of the reconstructed pseudo point cloud data is calculated by using the chamfer distance, and the expression is as follows: ； In the formula, represents the loss based on the reconstruction; represents the total number of point clouds; The expression of the total training loss is as follows: ； In the formula, denotes the total loss of the training process; , respectively denote the weight of the intermediate layer loss and the weight of the reconstructed loss. The updated pseudo point cloud feature encoder and the pseudo point cloud feature decoder are cascaded to reconstruct the data of the pseudo point cloud feature information and optimize the spatial distribution to generate optimized pseudo point cloud data; Based on the optimized pseudo point cloud data, combined with the binocular image features, a multi-modal feature fusion network is used for cross-modal fusion, and a camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is constructed by using a detection head; The camera multi-modal fusion 3D target detection model based on pseudo point cloud feature optimization is used for target detection on the binocular image to generate a target detection result.

2. The camera multi-modal fusion 3D object detection method based on pseudo point cloud feature optimization according to claim 1, characterized in that, Based on the binocular image data, binocular image features are extracted, and a stereo matching network is used for estimation and prediction, and combined with mapping into a three-dimensional space to generate pseudo point cloud data, which comprises the following steps: Based on the binocular image data, an image feature generation network is used to obtain binocular image features; Based on the binocular image features, 4D depth cost volumes are formed by connecting left and right image features, a stereo matching network is used to evaluate the credibility of each real depth value, and a prediction cost is generated; The prediction cost is converted into the probability of each real depth value, the weighted sum of all real depth values and their corresponding probabilities is calculated, the predicted depth value is calculated, and a smoothing loss function is used to supervise the difference between the real depth value and the predicted depth value; Combined with the camera parameters, the pixel points and their corresponding predicted depth values are mapped into a three-dimensional space to generate pseudo point cloud data; The expression of generating the pseudo point cloud data is as follows: ； ； In the formula, represents the pseudo point cloud data; represents the position on the X axis in the three-dimensional space; represents the position on the Y axis in the three-dimensional space; represents the position on the Z axis in the three-dimensional space; represents the physical distance between any two pixels on the X axis; represents the physical distance between any two pixels on the Y axis; represents the focal length of the camera; represents the position of the origin on the X axis in the pixel coordinate system; represents the position of the origin on the Y axis in the pixel coordinate system; represents the depth value corresponding to each pixel point in the pixel coordinate system, represents the coordinate of the corresponding pixel point on the X axis, represents the coordinate of the corresponding pixel point on the Y axis.

3. The camera multi-modal fusion 3D object detection method based on pseudo point cloud feature optimization according to claim 2, characterized in that, Based on the binocular image data, an image feature generation network is used to obtain binocular image features, which comprises the following steps: Based on the binocular image data, convolution layers and multiple residual modules are used for processing to obtain original image features output by each level of residual module; Adaptive average pooling layer is used to compress the image features output by the last residual module, and the original image features output by the selected remaining residual modules are spliced to generate binocular image features through convolution; The expression of generating binocular image features is: ； In the formula, represents binocular image features; represents an image feature generation network model; represents binocular image data.

4. The camera multi-modal fusion 3D object detection method based on pseudo point cloud feature optimization according to claim 1, characterized in that, The point cloud encoder and the pseudo point cloud encoder have the same structure and share weights; the encoding process specifically includes the following steps: Based on the point cloud data or the pseudo point cloud data, the SAB module in the PointNet++ backbone network is used to extract multi-scale point cloud features to generate weighted features of the point cloud data or the pseudo point cloud data; Based on the weighted features, the Transformer network architecture is used for self-attention reorganization to generate point cloud feature information or pseudo point cloud feature information; The expression of self-attention reorganization is: ； ； ； ； ； wherein, denotes the dimension of the feature vector; denotes the input feature vector; denotes the weighted feature obtained after the self-attention mechanism; denotes the output feature after enhancement; denotes the layer normalization function; denotes the output function; denotes the multi-layer perceptron; denotes the query; denotes the key; denotes the value matrix; denotes the weight of the query; denotes the weight of the key; denotes the matrix of the value matrix.

5. The camera multi-modal fusion 3D object detection method based on pseudo point cloud feature optimization according to claim 1, characterized in that, Based on the optimized pseudo point cloud data, combined with the binocular image features, a multi-modal feature fusion network is used for cross-modal fusion, and a camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization is constructed by using the detection head. The process specifically includes the following steps: Based on the optimized pseudo point cloud data, the feature generation network is used to extract the unordered optimized pseudo point cloud data voxel features as the optimized pseudo point cloud features. The expression of the pseudo point cloud features is: ； In the formula, represents the optimized pseudo point cloud feature information; represents the optimized pseudo point cloud data; represents a feature generation network; The optimized pseudo point cloud features and the binocular image features are input into the BEV feature generation network to form the BEV feature map of the pseudo point cloud modality and the BEV feature map of the binocular image. The BEV feature map of the pseudo point cloud modality and the BEV feature map of the binocular image are input into the multi-modal feature fusion network for initial fusion splicing, and the spliced features are fused twice by using the convolutional neural network to obtain the fusion features. The expression of the fusion features is: ； In the formula, represents a fusion feature of the pseudo point cloud feature and the binocular image feature; represents a multi-modal feature fusion network; represents a BEV feature map of the pseudo point cloud modality; represents a BEV feature map of the binocular image. The fusion features are used as the input of the detection head to train and construct a camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization.

6. A system for implementing the camera multi-modal fusion 3D object detection method based on pseudo point cloud feature optimization according to claim 1, characterized in that, It includes: A data acquisition module for acquiring binocular image data of a vehicle-mounted camera and point cloud data of a laser radar; A data generation module for extracting binocular image features based on binocular image data, and calculating a depth map combined with camera parameters to generate pseudo point cloud data in a three-dimensional space; A data optimization module for respectively encoding point cloud data and pseudo point cloud data to obtain point cloud feature information and pseudo point cloud feature information, and training a pseudo point cloud feature decoder supervised by the point cloud feature information to reconstruct and generate optimized pseudo point cloud data; A model construction module for combining the optimized pseudo point cloud data with binocular image features, using a multi-modal feature fusion network for cross-modal fusion, and constructing a camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization by using a detection head; A result generation module for performing target detection on binocular images by using the camera multi-modal fusion 3D target detection model based on the pseudo point cloud feature optimization to generate target detection results.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images