A multimodal three-dimensional target detection method and device for autonomous driving

Abstract

The present invention discloses a multimodal three-dimensional target detection method and device for autonomous driving, comprising the following steps: extracting image features from image data; extracting point cloud features from point cloud data; bidirectionally enhancing and fusing image features and point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features; respectively decoding the point cloud enhanced image features and the image enhanced point cloud features to obtain high-resolution image features and decoded point cloud features, fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features; sending the point cloud features of fused image features to a three-dimensional region proposal network and a refinement module for three-dimensional target detection. The present invention adopts an attention-based fusion method, in which the fusion module only focuses on useful features and suppresses noise, fully utilizing the complementarity of the two modalities for efficient fusion, and providing an effective solution for environmental perception of autonomous driving systems.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a multimodal three-dimensional target detection method and device for autonomous driving. Background Art

[0002] In recent years, with the continuous development of deep learning and computer vision technology, 3D object detection has become a key technology in the field of autonomous driving. According to the different input modal information, 3D object detection can be mainly divided into three categories: image, lidar and multimodal fusion. Images have rich color and texture attributes, but images cannot provide enough depth and geometric information and are easily affected by lighting. Image-based detection methods usually have low performance. LiDAR overcomes the problem of insufficient image depth information and provides accurate range view and geometric information, thus having excellent 3D detection performance. However, LiDAR also has many defects, such as low resolution, sparse point cloud information and lack of texture information. Sparse point clouds provide poor information in distant and occluded areas, and have poor detection effects on distant objects. Therefore, the sparsity and disorder of point clouds greatly limit the performance of LiDAR-based methods.

[0003] In order to solve the inherent limitations of a single modality and utilize the complementary characteristics of images and lidar, we hope to design a method based on multimodal fusion, hoping to provide a more complete three-dimensional environment perception through multi-source fusion, helping vehicles to better perceive the surrounding environment. However, fusing image and point cloud information is not easy because they have different feature representations. Summary of the invention

[0004] In order to solve the defects of the above-mentioned prior art, the present invention provides a multimodal three-dimensional target detection method and device for autonomous driving. The present invention uses a hybrid sampling method based on spatial distance and feature distance to sample the point cloud, collects more foreground points while also collecting sufficient background points, and designs an enhanced convolution module for image sampling, which enables the model to more accurately capture important features in image data; finally, a fusion module is designed to solve the problem of point cloud and image fusion. The bidirectional fusion module is a learnable process. The module uses an attention mechanism for adaptive weighted fusion, which fuses key information with high weight and discards redundant information with low weight.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: a multimodal three-dimensional target detection method for autonomous driving, comprising the following steps:

[0006] Extracting image features from image data;

[0007] Extract point cloud features from point cloud data;

[0008] Bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features;

[0009] Decoding the point cloud enhanced image features and the image enhanced point cloud features respectively to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features;

[0010] The point cloud features of the fused image features are sent to a three-dimensional region proposal network and a refinement module for three-dimensional object detection.

[0011] Furthermore, extracting image features from the image data includes the following sub-steps:

[0012] Convert the input image data X into output features Y, including,

[0013]

[0014] Wherein, Y represents the output feature Y, X represents the image data X, is the traditional convolution operation. represents the number of channels, and and Represent the height and width of the spatial feature respectively, R represents any image, C', W' and H' represent the channel, width and height of the image feature after the convolution operation respectively;

[0015] The output feature Y is converted into a feature vector Z, including,

[0016]

[0017] Wherein, Z represents the eigenvector Z, Represents a compression operation;

[0018] Convert the feature vector Z into output features , the output characteristics That is, the image features include:

[0019]

[0020] in, Represents the output characteristics , Represents incentive operation, Represents large-scale operation, refers to the ReLU function, Refers to the Sigmoid activation function, and are weight matrices, is the attention weight.

[0021] Furthermore, extracting point cloud features from point cloud data includes the following sub-steps:

[0022] At the first level set abstraction layer, single sampling is used to obtain the index of the sampling point in the original point cloud, wherein the single sampling method is the farthest point sampling based on the spatial distance, and the original point cloud refers to the initial point cloud data, including:

[0023]

[0024] in, Represents the original coordinate data of the point cloud, that is, , each Represents a point in the point cloud. The point set contains N points in total. Represents the index of the point in the original point cloud obtained by sampling the farthest point based on spatial distance;

[0025] Hybrid sampling is adopted from the second level set abstraction layer, including the index of the farthest point sampling based on spatial distance.

[0026]

[0027] in, It is a feature representation of point cloud. Represents the index according to sampling;

[0028] And including, using the farthest point sampling based on feature distance to obtain the index of the point selected for mixing,

[0029]

[0030] in, The index of the point selected for blending;

[0031] Among them, the mixed sampling is a mixture of the farthest point sampling based on spatial distance and the farthest point sampling based on feature distance. The set abstraction layer is four layers, the first layer is the farthest point sampling based on spatial distance, and the next three layers are based on mixed sampling.

[0032] Furthermore, obtaining the image features of point cloud enhancement includes the following sub-steps:

[0033] According to the original point set and image features, point-by-point image features are obtained, including:

[0034]

[0035]

[0036] in, Represents the original point set For any point in the image, its corresponding position is , M is the known projection matrix, is the bilinear interpolation function, yes The image features corresponding to the points, yes Image features of adjacent elements;

[0037] Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including:

[0038]

[0039]

[0040]

[0041]

[0042] in, is a learnable weight matrix, represents the global average pooling operation, represents the global maximum pooling operation, represents the convolution operation, represents the Sigmoid activation function, It is the result of the integration of different stages. represents the final fusion feature, is the attention weight of the point cloud feature, is the attention weight of the image feature;

[0043] The image features of point cloud enhancement are obtained based on the fusion features and image features, including:

[0044]

[0045] in, represents point-to-grid computation, is the final channel compression, It is the image feature of point cloud enhancement.

[0046] Furthermore, obtaining the point cloud features of image enhancement includes the following sub-steps:

[0047] According to the original point set and image features, point-by-point image features are obtained, including:

[0048]

[0049]

[0050] in, Represents the original point set For any point in the image, its corresponding position is , M is the known projection matrix, is the bilinear interpolation function, yes The image features corresponding to the points, yes Image features of adjacent elements;

[0051] Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including:

[0052]

[0053]

[0054]

[0055]

[0056] in, is a learnable weight matrix, represents the global average pooling operation, represents the global maximum pooling operation, represents the convolution operation, represents the Sigmoid activation function, It is the result of the integration of different stages. represents the final fusion feature, is the attention weight of the image feature, is the attention weight of the point cloud feature;

[0057] The point cloud features of image enhancement are obtained based on the fusion features and image features, including:

[0058]

[0059] in, represents point-to-grid computation, is the final channel compression, It is the point cloud feature of image enhancement.

[0060] A multimodal three-dimensional target detection device for autonomous driving, comprising:

[0061] Enhanced convolution module for extracting image features from image data;

[0062] Sampling and extraction module, used to extract point cloud features from point cloud data;

[0063] A fusion module, used for bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features;

[0064] A decoding module, used for respectively decoding the point cloud enhanced image features and the image enhanced point cloud features to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features;

[0065] The detection module is used to send the point cloud features of the fused image features into a three-dimensional region proposal network and a refinement module for three-dimensional target detection.

[0066] Furthermore, the enhanced convolution module includes:

[0067] Traditional convolution module, used to perform initial spatial feature extraction and dimensionality reduction on input image features;

[0068] Compression module for global average pooling using channels;

[0069] The excitation module is used to generate corresponding weights for each channel.

[0070] The scaling module is used to multiply the generated attention weights and features channel by channel to obtain the final output.

[0071] Furthermore, the fusion module includes:

[0072] The point cloud enhanced image module is used to find the correspondence between the point cloud and the image pixels in a point-by-point manner, and enhance the image features using the point cloud features to obtain the point cloud enhanced image features;

[0073] The image enhancement point cloud module is used to further enhance the point cloud features of the point cloud enhanced image features to obtain image enhanced point cloud features.

[0074] Furthermore, the decoding module includes:

[0075] Point cloud decoder, consisting of four feature propagation layers, is used to recover the spatial position and semantic information of point cloud features;

[0076] The image decoder consists of four parallel transposed convolutional layers to restore the image resolution.

[0077] In summary, the present invention has achieved the following technical effects:

[0078] The present invention uses the designed enhanced convolution module to dynamically capture important features in the image data; uses the designed distance-based and feature-based hybrid sampling method to sample the point cloud; performs bidirectional enhanced fusion of the image and point cloud features extracted at each layer, and further extracts features from the enhanced image and point cloud features respectively; decodes the image and point cloud respectively, and further enhances the point cloud features with the decoded image features; finally, the point cloud fused with the image features is sent to the 3D region proposal network and refinement module to achieve the final 3D target detection;

[0079] The present invention belongs to a multimodal 3D target detection algorithm, which can make full use of the advantages of images and original point clouds to achieve the complementarity of the two modalities. It is superior to single-modality-based methods in performance and is more widely applicable in practical computer vision application scenarios.

[0080] The encoding stage of the present invention is composed of an enhanced convolution module and a set abstraction based on mixed sampling, which can efficiently extract image and point cloud features and provide more useful feature representations for subsequent multimodal fusion;

[0081] The fusion module of the present invention directly fuses the original point cloud and image, effectively enhancing the features of the input modality without losing the point cloud and image information, and providing a more comprehensive feature representation for 3D object detection. The fusion module adaptively estimates the important features of the image and point cloud, and fully utilizes the complementarity of the two for efficient fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A schematic diagram of the network structure of the method of the present invention;

[0083] Figure 2 Schematic diagram of the enhanced convolution module of the present invention;

[0084] Figure 3 It is a point cloud enhanced image fusion module of the method of the present invention;

[0085] Figure 4 It is the image enhancement point cloud fusion module of the method of the present invention;

[0086] Fig.5a1 This is a visualization of the detection results of the model of the present invention on the data set. Figure 1 ;

[0087] Fig.5a2 The laser radar visualization result of the corresponding image of the present invention is shown in FIG. Figure 1 ;

[0088] Figure 5b1 This is a visualization of the detection results of the model of the present invention on the data set. Figure 2 ;

[0089] Figure 5b2The laser radar visualization result of the corresponding image of the present invention is shown in FIG. Figure 2 ;

[0090] Figure 5c1 This is a visualization of the detection results of the model of the present invention on the data set. Figure 3 ;

[0091] Figure 5c2 The laser radar visualization result of the corresponding image of the present invention is shown in FIG. Figure 3 . DETAILED DESCRIPTION

[0092] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0093] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

[0094] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0096] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0097] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0098] Embodiment:

[0099] like Figure 1 As shown, a multimodal three-dimensional target detection method for autonomous driving includes the following steps:

[0100] Step (1), extracting image features from image data;

[0101] Step (2), extracting point cloud features from the point cloud data;

[0102] Step (3), bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features;

[0103] Step (4), respectively decoding the point cloud enhanced image features and the image enhanced point cloud features to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features;

[0104] Step (5), sending the point cloud features of the fused image features into the 3D region proposal network and refinement module for 3D target detection.

[0105] In step (1), extracting image features from image data includes the following sub-steps: Figure 2 As shown,

[0106] (11) Convert the input image data X into output features Y, including:

[0107]

[0108] Wherein, Y represents the output feature Y, X represents the image data X, is the traditional convolution operation. represents the number of channels, and and Represent the height and width of the spatial feature respectively, Represents any image, C', W' and H' respectively represent the channel, width and height of the image features after the convolution operation;

[0109] in, The purpose is to perform initial spatial feature extraction and dimensionality reduction on the input image data X. Through the sequential combination of two 3×3 convolutional layers, two group normalization layers, and a ReLU activation layer, the input features are converted into output features Y. This processing method helps subsequent operations to extract richer feature representations, thereby obtaining more useful information and further improving model performance.

[0110] (12) Convert the output feature Y into a feature vector Z, including:

[0111]

[0112] Wherein, Z represents the eigenvector Z, represents the compression operation, which uses the global average pooling operation of the channel to directly compress the feature map containing global information into a 1×1×C' feature vector. The purpose of this is to convert each two-dimensional channel into a numerical value with a global receptive field, thereby shielding the spatial distribution information and better utilizing the correlation between channels;

[0113] (13) Convert the feature vector Z into output features , the output characteristics That is, the image features to be extracted in step 1, hereinafter referred to as image features, including:

[0114]

[0115] in, Represents the output characteristics , Represents incentive operation, Represents large-scale operation, refers to the ReLU function, Refers to the Sigmoid activation function, and are weight matrices, is the attention weight. First, after a full connection operation ( ), the weight matrix is After the ReLU operation, a second full connection operation is performed ( ), the weight matrix is In order to use In the information gathered during the compression operation, we use operation to fully capture the channel dependencies, and finally based on the correlation between feature channels, After activation, a weight is generated for each feature channel Used to indicate the importance of feature channels, the generated attention weights Multiply it channel by channel with feature Y to get the final output.

[0116] Extracting point cloud features from point cloud data in step (2) includes the following sub-steps:

[0117] (21) In the first level set abstraction layer, single sampling is used to obtain the index of the sampling point in the original point cloud, wherein the single sampling method is the farthest point sampling based on the spatial distance, and the original point cloud refers to the initial point cloud data, including:

[0118]

[0119] in, Represents the original coordinate data of the point cloud, that is, , each Represents a point in the point cloud. The point set contains N points in total. Represents the index of the point in the original point cloud obtained by sampling the farthest point based on spatial distance;

[0120] (22) Hybrid sampling is used starting from the second level set abstraction layer, including using the index obtained by sampling the farthest point based on spatial distance,

[0121]

[0122] in, It is a feature representation of point cloud. Represents the index according to sampling;

[0123] and including,

[0124] (23) The index of the point selected for mixing is obtained by sampling the farthest point based on the feature distance.

[0125]

[0126] in, Represents the index of the point selected by the mixture, and represents the summary of the indexes obtained by the two sampling releases. The spatial position of each point and the features of the point can be obtained through the index. The dataset stores the spatial position and feature description of the point together.

[0127] Among them, the mixed sampling is a mixture of the farthest point sampling based on spatial distance and the farthest point sampling based on feature distance. The set abstraction layer is four layers, the first layer is the farthest point sampling based on spatial distance, and the following three layers are all mixed sampling.

[0128] In this step, It can be information such as color, normal vector, density, intensity, etc., which is used to describe the properties of each point in the point cloud. By using these features, we can calculate the distance matrix , which records the distance between each pair of points in the point cloud. Then, we sample the farthest point according to the distance matrix to obtain the index according to the sampling .final is the index of the point selected for mixing. Mixed sampling is used to retain more positive points for positioning and ensure enough negative points for classification. The main mixed sampling is the farthest point sampling based on distance and the farthest point sampling based on feature distance, sampling half of the total number of points respectively. Mixed sampling is implemented starting from the second-level set abstraction layer, because the original point input of the first level cannot produce meaningful semantic features.

[0129] In step (3), obtaining the point cloud enhanced image features includes the following sub-steps: Figure 3 and Figure 4 As shown,

[0130] (31-1) Obtain point-by-point image features based on the original point set and image features, including:

[0131]

[0132]

[0133] in, Represents the original point set For any point in the image, its corresponding position is , M is the known projection matrix, is the bilinear interpolation function, yes The image features corresponding to the points, yes The image features of adjacent elements; the image features at the corresponding position p' are obtained using the bilinear interpolation function. The bilinear interpolation function can well obtain the image features around p'. Finally, we can get the total point-by-point image features corresponding to the point set P .

[0134] This part aims to obtain the features of the image at the corresponding position of the two-dimensional projection of the point cloud. The input information includes the original point set and image features (i=1,2,3,4), a point-by-point correspondence between the original three-dimensional point cloud and the two-dimensional position is established through the known projection matrix M.

[0135] (31-2) Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including:

[0136]

[0137]

[0138]

[0139]

[0140] in, is a learnable weight matrix, represents the global average pooling operation, represents the global maximum pooling operation, Represents a convolution operation, which consists of two 1×1 convolutions, two BatchNorm normalization layers, and a ReLU function layer. represents the Sigmoid activation function, It is the result of the integration of different stages. represents the final fusion feature, is the attention weight of the point cloud feature, is the attention weight of the image feature; in the fusion process, The purpose is to adjust the weight of the image feature so that the sum of its weight and the weight of the point cloud feature is 1. Otherwise, the model will pay too much attention to the point cloud feature and ignore the importance of the image feature, thus affecting the fusion effect. This ensures that during the fusion process, the two features can receive proper attention and balance, thereby achieving a better fusion effect.

[0141] First, through a 1×1 convolution operation, the input image features are transformed Point cloud features Reduce to the same number of channels to ensure that they have the same expressive power. The adjusted image features and point cloud features are then added to combine the information of the two different modalities to enhance the diversity and representation of the features. Next, global average pooling and global maximum pooling operations are performed on the added features to extract the global information of the input features and provide more comprehensive contextual information for subsequent feature enhancement and attention calculation. Subsequently, the series of convolutional layers is used to enhance the features in the channel dimension to improve the expressive power and discrimination of the features. After further dimensionality reduction, the Sigmoid activation function is used to normalize the weight mapping to the range of [0,1], from which we can obtain the attention weight of P2I-Fusion. , so that the model can adaptively focus on important features and suppress unimportant features. Finally, the image features and point cloud features are adjusted to the same channel, multiplied by their respective attention weights, and added element by element, and finally the fused feature representation regulated by the attention mechanism is obtained.

[0142] (31-3) The image features of point cloud enhancement are obtained based on the fusion features and image features, including:

[0143]

[0144] in, represents point-to-grid computation, is the final channel compression, It is the image feature of point cloud enhancement.

[0145] In order to integrate the features With image features To combine them, they need to be transformed into the same shape. The shape is B×C×N, The shape of is B×C×H×W, where B and C are the batch size and channel of the corresponding features respectively. The point-to-grid operation is based on the point-by-point correspondence, which scatters the point cloud features into the image grid and generates The projected point cloud features have the same shape. However, the projected positions of the points are rarely on the image grid, but usually between pixels. Therefore, we recover the features and the integer grid by applying bilinear interpolation to their neighboring point cloud features, thus obtaining a grid-like feature map. Then After being converted into a grid point cloud feature map and image features Finally, we input the concatenated features into a two-dimensional convolutional layer with a kernel size of 3 and a stride of 1 to compress the number of channels to The same channel is used to obtain the image features of point cloud enhancement.

[0146] Obtaining the point cloud features for image enhancement includes the following sub-steps:

[0147] (32-1) Obtain point-by-point image features based on the original point set and image features, including:

[0148]

[0149]

[0150] in, Represents the original point set For any point in the image, its corresponding position is , M is the known projection matrix, is the bilinear interpolation function, yes The image features corresponding to the points, yes Image features of adjacent elements;

[0151] (32-2) Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including:

[0152]

[0153]

[0154]

[0155]

[0156] in, is a learnable weight matrix, represents the global average pooling operation, represents the global maximum pooling operation, represents the convolution operation, represents the Sigmoid activation function, It is the result of the integration of different stages. represents the final fusion feature, is the attention weight of the image feature, is the attention weight of the point cloud feature;

[0157] (32-3) The point cloud features of image enhancement are obtained based on the fusion features and image features, including:

[0158]

[0159] in, represents point-to-grid computation, is the final channel compression, It is the point cloud feature of image enhancement.

[0160] The image features of the point cloud enhancement obtained in step 3 are designed to make full use of the rich semantic information of the image to enhance the point cloud features. Multiplying by the point-by-point image features, Multiply the point cloud features and add the features element by element and They have the same shape, i.e., B×C×N, so they can be directly concatenated to obtain the fused features. Finally, in order to make the module more flexible, the fused features are compressed to the same channel as the input point cloud features to obtain the final enhanced point cloud features. .

[0161] In step (4), after obtaining the enhanced image and point cloud features, further feature extraction is performed. After four layers of feature extraction, the image decoder uses four parallel transposed convolutional layers, each with a different step size, to restore the resolution of the image to ensure that the final feature map has the same resolution as the original image. Subsequently, these feature maps are combined in series and convolved to form a more representative feature map. , which contains rich image semantic information. Feature map It is not only used for the final feature representation, but also for enhancing point cloud features to generate more reliable 3D proposals. The point cloud feature decoder consists of four corresponding feature propagation layers to recover the spatial location and semantic information of point cloud features. In addition, we also introduce an image enhancement point cloud module to convert high-resolution image features into Further fusion with the decoded point cloud features to obtain the final enhanced point cloud features , to enhance the richness of point cloud features. Then, the fused features are fed into the 3D region proposal network for foreground point segmentation and 3D proposal generation, providing more reliable candidate boxes for subsequent target detection.

[0162] In step (5), the 3D region proposal network generated in step 4 generates proposals centered on each point, and further improves and refines the region of interest through the refinement module to finally complete the bounding box prediction and classification tasks. The refinement module consists of three set abstraction layers and two detection heads. We randomly select 512 points from the proposals as feature descriptions of the 3D region of interest. For those proposals with less than 512 points, we use simple point sampling and replace to reach the number of 512 points. This method effectively handles objects of different sizes and shapes and improves the stability and performance of the model.

[0163] The present invention first extracts features from the image and the original point cloud respectively, and then performs bidirectional enhancement fusion on the image and point cloud features extracted in sequence. The fusion module uses the attention mechanism to pay more attention to the useful features of the two modalities that complement each other and reduce noise; then the enhanced image and point cloud features are further extracted, and there are a total of four layers of feature extraction links; the image decoder uses four parallel transposed convolutional layers to restore the resolution of the image, and the point cloud feature decoder is composed of four corresponding feature propagation layers. In addition, the final image decoding feature map is also used for the last enhancement of the point cloud features; finally, the enhanced image fused point cloud features are input into the three-dimensional region proposal network and the refinement module to realize the final three-dimensional target detection. The present invention adopts an attention-based fusion method. The fusion module only pays attention to useful features and suppresses noise, making full use of the complementarity of the two modalities for efficient fusion, and provides an effective solution for the environmental perception of the autonomous driving system.

[0164] In addition, the present invention uses a multi-task loss function to jointly optimize the entire network, and the total loss function is as follows:

[0165]

[0166] in, and They represent the loss functions of the 3D region proposal network stage and the refinement network module, respectively, which adopt similar optimization objectives.

[0167] For outdoor scenes, the number of foreground points is usually much smaller than the number of background points. We use focal loss as the classification loss to balance positive and negative samples, which is calculated as:

[0168]

[0169] in, is the probability that the point belongs to the true class, =0.25, =2.0.

[0170] For the bounding box, the network needs to regress the center point of the object ,size and direction Since the target center height The range is relatively small, so we directly use smooth L1 loss to calculate its offset from the true value. The size of is also used smooth L1 loss. , and , we adopt bin-based regression loss. For each foreground point, we divide its surrounding area into several bins. The bin-based loss first predicts where the center point falls. Then the residual in the bin The specific formula for regression is as follows.

[0171]

[0172] in, and They represent cross entropy loss and smooth L1 loss respectively. and Represents the true value of bins and residuals.

[0173] In addition to the above losses, image segmentation loss is introduced to address the inconsistency problem between classification and positioning confidence and to ensure the consistency between the confidence scores of image and point cloud predictions. , multimodal consistency loss and consistency enforcement loss .

[0174] In summary, the total loss in the RPN stage is the sum of the above losses. for The weight of is set to 5. For similar optimization , not including , .

[0175] .

[0176] A multimodal three-dimensional target detection device for autonomous driving, comprising:

[0177] Enhanced convolution module for extracting image features from image data;

[0178] Sampling and extraction module, used to extract point cloud features from point cloud data;

[0179] A fusion module, used for bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features;

[0180] A decoding module, used for respectively decoding the point cloud enhanced image features and the image enhanced point cloud features to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features;

[0181] The detection module is used to send the point cloud features of the fused image features into a three-dimensional region proposal network and a refinement module for three-dimensional target detection.

[0182] The enhanced convolution module includes:

[0183] Traditional convolution module, used to perform initial spatial feature extraction and dimensionality reduction on input image features;

[0184] Compression module for global average pooling using channels;

[0185] The excitation module is used to generate corresponding weights for each channel.

[0186] The scaling module is used to multiply the generated attention weights and features channel by channel to obtain the final output.

[0187] The fusion module includes:

[0188] The point cloud enhanced image module is used to find the correspondence between the point cloud and the image pixels in a point-by-point manner, and enhance the image features using the point cloud features to obtain the point cloud enhanced image features;

[0189] The image enhancement point cloud module is used to further enhance the point cloud features of the point cloud enhanced image features to obtain image enhanced point cloud features.

[0190] The decoding module comprises:

[0191] Point cloud decoder, consisting of four feature propagation layers, is used to recover the spatial position and semantic information of point cloud features;

[0192] The image decoder consists of four parallel transposed convolutional layers to restore the image resolution.

[0193] In this example, the above method and model are applied to the KITTI dataset and compared with the current mainstream image-based, point cloud-based single-modal methods and image and point cloud fusion multi-modal methods. The dataset has three categories: car, pedestrian, and cyclist, and the difficulty level is divided into easy, medium, and difficult. The overall performance comparison results are shown in Table 1. Refers to the average accuracy of 3D detection. '-' means that the result is not reported or the source code is not open. "L" and "I" represent point cloud and image respectively, and the best results are bolded.

[0194] Table 1 Overall performance comparison results

[0195]

[0196] As can be seen from Table 1, in the mainstream car category, the proposed method surpasses the method using only LiDAR. Compared with the framework PointRCNN, it is improved by 6.95% on medium difficulty. These experimental results prove the effectiveness of our proposed multimodal fusion method. Compared with the method using only image and LiDAR fusion, the proposed method shows the best and almost the same performance at all difficulty levels. For the pedestrian category, the proposed method still achieves close or even better performance. For cyclists, the proportion in the KITTI dataset is about 4.67%, which is much smaller than the proportion of pedestrians (12.87%) and cars (82.46%). Compared with the method that uses the GT Sampling strategy to construct more instances [9]

[12] , the proposed method has lower performance in this category. Due to the small amount of training data, most state-of-the-art methods do not provide data comparisons for cyclists and pedestrians, and only test their models on the car category because of its large data volume and complex scenes. It is worth noting that the proposed method outperforms PointRCNN at all difficulty levels for cyclists, which further proves the effectiveness of the proposed method.

[0197] The present invention selects three pictures and the corresponding point cloud visualization model detection results, such as Fig.5a1 ~c2. The truth box is represented by green, the detection results fused by direct splicing are shown in blue, and the results fused by the Att-Fusion module designed by the present invention are shown in red. The display results of each row are RGB images on the left and radar images on the right. For distant targets, due to the sparse radar point cloud, image information can provide richer texture information. In areas with insufficient light and dark occlusion, there is less useful information in the image and there is more noise, while radar can provide more reliable distance and geometric information. The Att-Fusion module designed by the present invention can selectively focus on the most relevant features in each modality, thereby reducing the impact of redundant information. In contrast, directly splicing the two modalities will introduce more noise. By observing Fig.5a1 In the radar map visualized in ~c2, the detection effect of the present invention on targets far away and with insufficient illumination is better than that of the direct splicing method, and is closer to the true value frame. This once again illustrates the effectiveness of the present invention.

[0198] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal three-dimensional target detection method for autonomous driving, characterized by: The following steps are included: Extracting image features from image data; Extract point cloud features from point cloud data; Bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features; Decoding the point cloud enhanced image features and the image enhanced point cloud features respectively to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features; Sending the point cloud features of the fused image features into a 3D region proposal network and a refinement module to perform 3D object detection; Extracting image features from image data includes the following sub-steps: Convert the input image data X into output features Y, include, Y=F tr (X),X∈R W×H×C ,Y∈R W′×H′×C′ Wherein, Y represents the output feature Y, X represents the image data X, and F tr It is a traditional convolution operation. C represents the number of channels, while W and H represent the height and width of the spatial features respectively. R represents any image. C', W' and H' represent the channel, width and height of the image features after the convolution operation respectively. The output feature Y is converted into a feature vector Z, including, Z=F sq (Y) Where Z represents the eigenvector Z, F sq Represents a compression operation; Convert the feature vector Z into output features The output characteristics That is, the image features include: in, Represents the output characteristics F ex represents the incentive operation, F scale represents the scale operation, δ refers to the ReLU function, σ refers to the Sigmoid activation function, W1 and W2 are both weight matrices, and ω is the attention weight; Extracting point cloud features from point cloud data includes the following sub-steps: At the first level set abstraction layer, single sampling is used to obtain the index of the sampling point in the original point cloud, wherein the single sampling method is the farthest point sampling based on the spatial distance, and the original point cloud refers to the initial point cloud data, including: idx d =FPS(xyz,N / 2) Among them, xyz represents the original coordinate data of the point cloud, that is, xyz = {p1, p2, ..., p N }, each p i Represents a point in the point cloud. The point set contains N points in total. idx d Represents the index of the point in the original point cloud obtained by sampling the farthest point based on spatial distance; Since the second level set abstraction layer, mixed sampling is used, including using the farthest point sampling based on spatial distance to obtain the index, idx f =FPS(matrix(xyz,f),N / 2 Among them, f is a feature representation of point cloud, idx f Represents the index according to sampling; And including, using the farthest point sampling based on feature distance to obtain the index of the point selected for mixing, idx=idx d ||idx f Among them, idx represents the index of the point selected for mixing; The mixed sampling is a mixture of the farthest point sampling based on spatial distance and the farthest point sampling based on feature distance. The set abstraction layer is four layers, the first layer is the farthest point sampling based on spatial distance, and the following three layers are based on mixed sampling. Obtaining point cloud enhanced image features includes the following sub-steps: According to the original point set and image features, point-by-point image features are obtained, including: p′=Mp Where p represents any point in the original point set P, its corresponding position in the image is p′, M is the known projection matrix, is the bilinear interpolation function, is the image feature corresponding to point p, is the image feature of the adjacent elements of p′; Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including: S2=Conv(GAP(S1))+Conv(GMP(S1)) oh p2I =σ(w3S2) Among them, w i (i=1,2,3,4,5) is a learnable weight matrix, GAP represents the global average pooling operation, GMP represents the global maximum pooling operation, Conv represents the convolution operation, σ represents the Sigmoid activation function, S j (j=1,2,3) are the fusion results at different stages, S3 represents the final fusion feature, ω p2I is the attention weight of the point cloud feature, 1-ω p2I is the attention weight of the image feature; The image features of point cloud enhancement are obtained based on the fusion features and image features, including: Among them, P2G represents point-to-grid calculation, Conv2D is the final channel compression, and F EI is the image feature enhanced by point cloud, Represents image features; Obtaining the point cloud features for image enhancement includes the following sub-steps: According to the original point set and image features, point-by-point image features are obtained, including: p′=Mp Where p represents any point in the original point set P, its corresponding position in the image is p′, M is the known projection matrix, is the bilinear interpolation function, is the image feature corresponding to point p, F EI (p′) is the image feature of the neighboring elements of p′; Calculate the attention weight, and obtain the fusion feature according to the point-by-point image feature and the point cloud feature, including: S2=Conv(GAP(S1))+Conv(GMP(S1)) oh I2P =σ(w3S2) Among them, w i (i=1,2,3,4,5) is a learnable weight matrix, GAP represents the global average pooling operation, GMP represents the global maximum pooling operation, Conv represents the convolution operation, σ represents the Sigmoid activation function, S j (j=1,2,3) are the fusion results at different stages, S3 represents the final fusion feature, ω I2P is the attention weight of the image feature, 1-ω I2P is the attention weight of the point cloud feature; The point cloud features of image enhancement are obtained based on the fusion features and image features, including: Among them, Conv2D is the final channel compression, F eP It is the point cloud feature of image enhancement.

2. A multimodal three-dimensional target detection device for autonomous driving, characterized in that: A multimodal three-dimensional target detection method for autonomous driving as claimed in claim 1, comprising: Enhanced convolution module for extracting image features from image data; Sampling and extraction module, used to extract point cloud features from point cloud data; A fusion module, used for bidirectionally enhancing and fusing the image features and the point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features; A decoding module, used for respectively decoding the point cloud enhanced image features and the image enhanced point cloud features to obtain high-resolution image features and decoded point cloud features, and further fusing the high-resolution image features with the decoded point cloud features to obtain point cloud features of fused image features; The detection module is used to send the point cloud features of the fused image features into a three-dimensional region proposal network and a refinement module for three-dimensional target detection.

3. The multimodal three-dimensional target detection device for autonomous driving according to claim 2, characterized in that: The enhanced convolution module includes: Traditional convolution module, used to perform initial spatial feature extraction and dimensionality reduction on input image features; Compression module for global average pooling using channels; The excitation module is used to generate corresponding weights for each channel. The scaling module is used to multiply the generated attention weights and features channel by channel to obtain the final output.

4. The multimodal three-dimensional target detection device for autonomous driving according to claim 2, characterized in that: The fusion module includes: The point cloud enhanced image module is used to find the correspondence between the point cloud and the image pixels in a point-by-point manner, and enhance the image features using the point cloud features to obtain the point cloud enhanced image features; The image enhancement point cloud module is used to further enhance the point cloud features of the point cloud enhanced image features to obtain image enhanced point cloud features.

5. The multimodal three-dimensional target detection device for autonomous driving according to claim 2, characterized in that: The decoding module comprises: Point cloud decoder, consisting of four feature propagation layers, is used to recover the spatial position and semantic information of point cloud features; The image decoder consists of four parallel transposed convolutional layers to restore the image resolution.

Images