Three-dimensional target detection method, electronic device, and medium

Abstract

The application discloses a three-dimensional target detection method, electronic equipment and medium, and belongs to the field of computer vision. The three-dimensional target detection method comprises the following steps: acquiring scene data of a target scene, wherein the scene data comprises first point cloud data collected by a laser radar and multi-view images collected by a camera at the same time; generating a first bird's eye view according to the first point cloud data, and generating a second bird's eye view according to the multi-view images; inputting the first bird's eye view and the second bird's eye view into an encoding-decoding network based on an attention mechanism, wherein the encoding-decoding network comprises an encoder and a decoder; guiding the encoder to model the first bird's eye view based on the second bird's eye view, obtaining an enhanced vector representation, and predicting the enhanced vector representation based on the decoder, thereby obtaining a target detection result of the target scene. The application solves the technical problem of low accuracy of a target detection method based on multi-modal fusion.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and particularly relates to a three-dimensional target detection method, electronic device, and medium. Background Technology

[0002] With the development of autonomous driving technology, 3D object detection has become a core task for environmental perception in autonomous driving systems. Traditional 3D object detection mainly relies on point cloud data provided by LiDAR, with the representative method CenterPoint locating targets by generating a bird's-eye view (BEV) from voxelized point clouds, achieving a good balance between accuracy and speed. However, 3D object detection based solely on point cloud data has the following shortcomings: 1. Point cloud density decays rapidly with distance, resulting in sparse information about distant targets; 2. It cannot provide rich semantic information, such as object texture and color; 3. Robustness is reduced under occlusion conditions.

[0003] To address these issues, multimodal fusion-based target detection methods have been developed. These methods incorporate image information into the 3D target detection process, leveraging the rich semantic features of the image to compensate for the limitations of point cloud data in identifying distant targets and detailed features, thereby improving the overall performance of 3D target detection. However, current multimodal fusion-based target detection methods often involve simple feature stitching and stacking. The fused features fail to fully utilize the complementary advantages of image semantics and point cloud data, ultimately resulting in low accuracy and failing to meet the high-precision detection requirements of practical applications. Summary of the Invention

[0004] This invention provides a three-dimensional target detection method, electronic device, and medium to solve the technical problem of low accuracy in target detection methods based on multimodal fusion.

[0005] In a first aspect, embodiments of the present invention provide a three-dimensional target detection method, comprising: acquiring scene data of a target scene, the scene data including first point cloud data acquired by a lidar and multi-view images acquired by a camera at the same time; generating a first bird's-eye view based on the first point cloud data, and generating a second bird's-eye view based on the multi-view images; inputting the first bird's-eye view and the second bird's-eye view into an attention-based encoder-decoder network, the encoder-decoder network including an encoder and a decoder; guiding the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation, and the decoder making predictions based on the enhanced vector representation to obtain a target detection result of the target scene.

[0006] In conjunction with the first aspect, in some embodiments, generating a first bird's-eye view based on the first point cloud data and generating a second bird's-eye view based on the multi-view images includes: a first feature extraction network performing voxel-level feature encoding on the first point cloud data to obtain the first bird's-eye view; a second feature extraction network performing feature extraction on the multi-view images to obtain corresponding multi-view image features; and using the intrinsic and extrinsic parameter matrices of the camera to map the multi-view image features to the bird's-eye view space to obtain the second bird's-eye view.

[0007] In conjunction with the first aspect, in some embodiments, the first feature extraction network is a three-dimensional convolutional neural network based on voxel networks, and the second feature extraction network is a two-dimensional convolutional neural network.

[0008] In conjunction with the first aspect, in some embodiments, the step of mapping the multi-view image features to a bird's-eye view space using the intrinsic and extrinsic parameter matrices of the camera to obtain the second bird's-eye view image includes: using the intrinsic and extrinsic parameter matrices of the camera to convert the pixels in the multi-view image features into three-dimensional coordinate points in the camera coordinate system; using the extrinsic parameter matrix to convert the three-dimensional coordinate points to the vehicle coordinate system to obtain second point cloud data; and projecting the second point cloud data onto the bird's-eye view space to obtain the second bird's-eye view image.

[0009] In conjunction with the first aspect, in some embodiments, the encoder includes a first input layer and an M-layer encoder layer, where M is an integer of 1. The step of guiding the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation includes: the first input layer converting the first bird's-eye view into a first vector representation and converting the second bird's-eye view into a second vector representation; the first encoder layer in the M-layer encoder layer performing a linear transformation on the first vector representation using a first weight matrix to obtain a first key matrix, performing a linear transformation on the first vector representation using a second weight matrix to obtain a first value matrix, and performing a linear transformation on the second vector representation using a third weight matrix to obtain a first query matrix; the first encoder layer performing cross-attention interaction based on the first key matrix, the first value matrix, and the first query matrix to obtain a fused vector representation; and the M-1 encoder layers in the M-layer encoder layer (excluding the first encoder layer) sequentially performing fusion processing on the fused vector representation output by the previous encoder layer to obtain the enhanced vector representation.

[0010] In conjunction with the first aspect, in some embodiments, the decoder makes predictions based on the enhanced vector representation to obtain the target detection result of the target scene, including: filtering out potential target regions in the second bird's-eye view; mapping the potential target regions to three-dimensional candidate centers; the decoder uses the three-dimensional candidate centers as query vectors to perform regression prediction on the target region in the enhanced vector representation, and outputs the target detection result.

[0011] In conjunction with the first aspect, in some embodiments, the step of filtering out potential target areas in the second bird's-eye view includes: calculating the response intensity of each grid cell in the second bird's-eye view; selecting each grid cell in the second bird's-eye view whose response intensity is greater than a preset intensity threshold as the potential target area, or filtering out the K grid cells with the largest response intensity in the second bird's-eye view as the potential target area.

[0012] According to a second aspect of the present invention, a three-dimensional target detection device is provided, comprising: a data acquisition unit for acquiring scene data of a target scene, wherein the scene data is first point cloud data acquired by a lidar and multi-view images acquired by a camera; an image generation unit for generating a first bird's-eye view based on the first point cloud data and generating a second bird's-eye view based on the multi-view images; an input unit for inputting the first bird's-eye view and the second bird's-eye view into an attention-based encoder-decoder network, wherein the encoder-decoder network includes an encoder and a decoder; and an encoder-decoder unit for guiding the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation, wherein the decoder makes a prediction based on the enhanced vector representation to obtain a target detection result of the target scene.

[0013] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional target detection method as described in any one of the embodiments of claims 1-7.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the three-dimensional target detection method described in any embodiment.

[0015] Through one or more embodiments provided by the present invention, at least the following technical effects or advantages are achieved: By generating a first bird's-eye view from first point cloud data and a second bird's-eye view from multi-view images, the LiDAR point cloud and multi-view images are uniformly converted into a bird's-eye view. The first and second bird's-eye views are input into an attention-based encoder-decoder network. The encoder, guided by the second bird's-eye view, models the first bird's-eye view, obtaining an enhanced vector representation. The decoder then makes predictions based on this enhanced vector representation, yielding the target detection result for the target scene. This achieves feature modeling and enhancement of the first bird's-eye view obtained from point cloud data using the second bird's-eye view obtained from multi-view images. Image semantics directly influence the point cloud feature modeling process during the encoding stage, enabling precise alignment and deep fusion of heterogeneous multimodal data in the same view space. By fully utilizing image semantics to compensate for missing point cloud features, the accuracy of multimodal fusion-based target detection methods can be improved, thus meeting the high-precision detection requirements of practical applications. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a three-dimensional target detection method according to some embodiments of the present invention is shown; Figure 2 A schematic diagram of the structure of a three-dimensional target detection device according to some embodiments of the present invention is shown; Figure 3 A schematic diagram of the structure of an electronic device according to some embodiments of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] This invention provides a three-dimensional target detection method that can be applied to scenarios such as autonomous driving of vehicles, intelligent traffic monitoring, autonomous navigation and obstacle avoidance of robots, and three-dimensional mapping and monitoring of unmanned aerial vehicles. Figure 1 A flowchart of a three-dimensional target detection method according to some embodiments of the present invention is shown. For example... Figure 1As shown, the three-dimensional target detection method includes the following steps S101 to S104.

[0020] Step S101: Obtain scene data of the target scene, the scene data including first point cloud data collected by lidar at the same time and multi-view images collected by camera.

[0021] In some embodiments, multi-view images are acquired through multiple cameras mounted on the vehicle body. In autonomous driving scenarios, these multiple cameras are installed at different locations on the vehicle equipped with an autonomous driving system. For example, the vehicle may have front-view, side-front-view, side-rear-view, and rear-view cameras. Each camera acquires images of the target scene from different perspectives to obtain multi-view images, achieving comprehensive coverage of the target scene, avoiding blind spots, and providing rich semantic input for the subsequent generation of a second bird's-eye view. In autonomous driving scenarios, the acquired first point cloud data includes the position and shape information of surrounding vehicles, pedestrians, roads, etc.

[0022] Step S102: Generate a first bird's-eye view based on the first point cloud data, and generate a second bird's-eye view based on the multi-view images.

[0023] In some embodiments, in step S102, generating a first bird's-eye view based on the first point cloud data may include: a first feature extraction network performing voxel-level feature encoding on the first point cloud data to obtain the first bird's-eye view.

[0024] In some embodiments, the first feature extraction network can be a three-dimensional convolutional neural network based on voxel networks. For example, in the CenterPoint model, the three-dimensional convolutional neural network based on voxel networks can be a convolutional neural network based on the VoxelNet structure. The VoxelNet-based convolutional neural network performs voxel-level feature encoding on the first point cloud data to obtain a first bird's-eye view, which may include: voxelizing the first point cloud data based on a set voxel size and number to divide the three-dimensional space containing the first point cloud data into multiple three-dimensional voxel grids; then, encoding each voxel grid to convert the first point cloud data into the first bird's-eye view. It can be understood that by voxelizing the first point cloud data, the first point cloud data can be encoded into individual three-dimensional voxel grids. Each voxel grid contains some point cloud information, such as the number of points and the average normal vector.

[0025] In some embodiments, in step S102, generating a second bird's-eye view based on multi-view images may include: a second feature extraction network extracting features from the multi-view images to obtain corresponding multi-view image features; and using the intrinsic and extrinsic parameter matrices of the camera to map the multi-view image features to the bird's-eye view space to obtain the second bird's-eye view.

[0026] In some embodiments, the second feature extraction network is a two-dimensional convolutional neural network, such as a ResNet50 network, which is a 50-layer version of a residual network. Multi-view images are multiple images captured by cameras at different locations. The second feature extraction network extracts features from the multi-view images, including: preprocessing each image separately to ensure that each preprocessed image meets the input requirements of the ResNet50 network. For example, each preprocessed image is input into the ResNet50 network, and the network extracts image features layer by layer to obtain the multi-view image features. It should be understood that in actual implementation, networks such as ResNet101, ResNet34, and EfficientNet-B4 / B5 can be used instead of the ResNet50 network.

[0027] In some embodiments, mapping multi-view image features to a bird's-eye view space using the intrinsic and extrinsic parameter matrices of the camera to obtain a second bird's-eye view may include: for the image features extracted from each image, using the intrinsic and extrinsic parameter matrices of the camera that acquired the image, converting the pixels in the extracted image features from the image into three-dimensional coordinate points in the camera coordinate system; using the extrinsic parameter matrix of the camera that acquired the image, transforming the three-dimensional coordinate points into the vehicle coordinate system to obtain the point cloud data corresponding to the image; stitching together the point cloud data corresponding to all images to obtain the second point cloud data; and projecting the second point cloud data into the bird's-eye view space to obtain the second bird's-eye view. It is understood that the projection of the second point cloud data in the vehicle coordinate system into the bird's-eye view space can be accomplished through high compression or a planar assumption to obtain the second bird's-eye view.

[0028] Step S103: Input the first bird's-eye view and the second bird's-eye view into the attention-based encoder-decoder network. The encoder-decoder network includes an encoder and a decoder.

[0029] In some embodiments, the attention-based encoder-decoder network is a Transformer architecture encoder-decoder network, wherein the encoder of the encoder-decoder network includes a first input layer and an encoding layer, and the decoder of the encoder-decoder network includes a second input layer, a decoding layer and an output layer.

[0030] The first input layer is used to convert the first bird's-eye view into a first vector representation, and to convert the second bird's-eye view into a second vector representation.

[0031] In some embodiments, the encoding layer is an M-layer encoder layer, meaning it is composed of M stacked encoder layers, where M is an integer of 1. For example, the encoding layer is composed of 4 to 12 stacked encoder layers. Exemplarily, the encoding layer is configured to consist of 4, 6, 8, or 12 stacked encoder layers. The decoding layer is an N-layer decoder layer, meaning it is composed of N stacked decoder layers, where N is an integer of 1. For example, the decoding layer is composed of 4 to 12 stacked decoder layers. Exemplarily, the decoding layer is configured to consist of 4, 6, 8, or 12 stacked decoder layers. In some embodiments, both the encoding and decoding layers can be configured to have 6 layers.

[0032] Understandably, each encoder layer consists of two sub-layer connections: the first is an attention sub-layer, and the second is a feedforward fully connected sub-layer, with each sub-layer followed by a normalization layer and a residual connection. The decoder layer consists of N stacked decoder layers. Each decoder layer consists of three sub-layer connections: the first is an attention sub-layer, the second is a multi-head attention sub-layer (encoder to decoder), and the third is a feedforward fully connected layer. Each sub-layer is followed by a normalization layer and a residual connection. The output layer includes a linear layer and a softmax layer: the linear layer transforms the vector output from the decoder layer into the final output dimension. The softmax layer transforms the output of the linear layer into a probability distribution for the final 3D target regression prediction.

[0033] Step S104: Based on the second bird's-eye view, guide the encoder to model the first bird's-eye view to obtain an enhanced vector representation. The decoder makes predictions based on the enhanced vector representation to obtain the target detection results of the target scene.

[0034] In some embodiments, the encoder modeling of the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation may include: a first input layer converting the first bird's-eye view into a first vector representation and converting the second bird's-eye view into a second vector representation; a first encoder layer in the M-layer encoder layer performing a linear transformation on the first vector representation using the first weight matrix of the first encoder layer to obtain a first key matrix, performing a linear transformation on the first vector representation using the second weight matrix of the first encoder layer to obtain a first value matrix, and performing a linear transformation on the second vector representation using the third weight matrix of the first encoder layer to obtain a first query matrix; the first encoder layer performing cross-attention interaction based on the first key matrix, the first value matrix, and the first query matrix in the first encoder layer to output a fused vector representation; and the M-1 encoder layers in the M-layer encoder layer, excluding the first encoder layer, sequentially performing fusion processing on the fused vector representation output by the previous encoder layer to obtain an enhanced vector representation.

[0035] Understandably, the attention sublayer of the first encoder layer uses the first and second weight matrices from the first encoder layer to perform a linear transformation on the first vector representation, resulting in the first key matrix and the first value matrix. Similarly, the first encoder layer uses the third weight matrix from the first encoder layer to perform a linear transformation on the second vector representation, resulting in the first query matrix. Specifically, the first key matrix, the first value matrix, and the first query matrix can be calculated using the following formulas: K1=F pc_bev ×W K1 V1=F pc_bev ×W V1 Q 1= F img_bev ×W Q1 Among them, W K1 W is the first weight matrix in the first encoder layer. V1 W is the second weight matrix in the first encoder layer. Q1 F is the third weight matrix in the first encoder layer. pc_bev F is the first vector representation corresponding to the first bird's-eye view. img_bev The second vector representation is the second bird's-eye view, and K1, V1, and Q1 correspond to the first key matrix, the first value matrix, and the first query matrix.

[0036] In some embodiments, the first encoder layer performs cross-attention interaction based on the first key matrix, the first value matrix, and the first query matrix obtained from the first encoder layer, and outputs a fused vector representation, which may include the following steps 1-2: Step 1: The attention sublayer in the first encoder layer performs a dot product operation on the transpose of the first query matrix and the first key matrix to obtain the original attention score matrix; the original attention score matrix is ​​scaled to obtain a scaled score matrix, with the scaling factor being the square root of the feature dimension; the scaled score matrix is ​​normalized to obtain the attention weight matrix; the first value matrix is ​​weighted and summed using the attention weight matrix to obtain the cross-attention output.

[0037] Step 2: The feedforward fully connected sublayer in the first encoder layer adds the cross-attention output to the original input features (i.e., the first vector representation) of the first encoder layer element by element to achieve residual connection; and performs layer normalization on the residual connection result to obtain the fused vector representation output by the first encoder layer.

[0038] In the above process, the calculation of the cross-attention output is referenced by the following formula:

[0039] in, As scaling factors, K1, V1, and Q1 correspond to the first key matrix, the first value matrix, and the first query matrix, respectively. The calculation results... This is the cross-attention output of the attention sublayer in the first encoder layer.

[0040] After the first encoder layer outputs the fused vector representation, any one of the following first, second, and third implementation methods can be used to enable the M-1 encoder layers (excluding the first encoder layer) in the M-layer encoder layer to sequentially perform fusion processing on the fused vector representation output by the previous encoder layer to obtain the enhanced vector representation.

[0041] In the first embodiment, the M-1 encoder layers (excluding the first encoder layer) in the M-1 encoder layer sequentially perform fusion processing on the fusion vector representation output by the previous encoder layer. This may include: each encoder layer in the M-1 encoder layer performs encoding based on the fusion vector representation output by the previous encoder layer using a self-attention mechanism to obtain the fusion vector representation output by the current encoder layer, and so on, until the last encoder layer in the M-1 encoder layer outputs the fusion vector representation, and the fusion vector representation output by the last encoder layer is used as the enhanced vector representation.

[0042] It is understandable that each encoder layer in the M-1 encoder layers (excluding the first encoder layer) performs self-attention-based encoding based on the fused vector representation output by the previous encoder layer. The output fused vector representation of this encoder layer can be obtained through the following steps: The i-th encoder layer uses the first weight matrix in the i-th encoder layer to perform a linear transformation on the fused vector representation output by the (i-1)-th encoder layer to obtain the i-th key matrix; it then uses the second weight matrix in the i-th encoder layer to perform a linear transformation on the fused vector representation output by the (i-1)-th encoder layer to obtain the i-th value matrix; finally, it uses the third weight matrix in the i-th encoder layer to perform a linear transformation on the fused vector representation output by the (i-1)-th encoder layer to obtain the i-th query matrix; the i-th encoder layer then performs self-attention interaction based on the i-th key matrix, the i-th value matrix, and the i-th query matrix to obtain the fused vector representation output by the i-th encoder layer. It should be noted that i ranges from 2 to M, where M is the number of encoder layers.

[0043] Understandably, the i-th encoder layer performs self-attention interaction based on the i-th key matrix, i-th value matrix, and i-th query matrix to obtain the fused vector representation output by the i-th encoder layer. This can include: the attention sub-layer in the i-th encoder layer performing: a dot product operation on the transpose of the i-th query matrix and the i-th key matrix to obtain the original attention score matrix; scaling the original attention score matrix to obtain a scaled score matrix, with the scaling factor being the square root of the feature dimension; normalizing the scaled score matrix to obtain the attention weight matrix; and using the attention weight matrix to perform a weighted summation on the i-th value matrix to obtain the self-attention output. The feedforward fully connected sub-layer in the i-th encoder layer adds the self-attention output element-wise to the original input features of the encoder layer to achieve residual connection, and performs layer normalization on the residual connection result to obtain the fused vector representation output by the i-th encoder layer.

[0044] It is understandable that when encoding based on the fused vector representation output by the previous encoder layer using a self-attention mechanism, the attention sub-layer in the i-th encoder layer uses the first, second, and third weight matrices of the i-th encoder layer to perform a linear transformation on the fused vector representation output by the (i-1)-th encoder layer, corresponding to obtaining the i-th key matrix, the i-th value matrix, and the i-th query matrix, as shown in the following formula: K i =X bev_i-1 ×W Ki V i =X bev_i-1 ×W Vi Q i= X bev_i-1 ×W Qi Among them, X bev_i-1 W is the fused vector representation output by the previous encoder layer (i.e., the (i-1)th encoder layer). Ki、 W Vi、 W Qi These correspond to the first, second, and third weight matrices in the i-th encoder layer (i.e., this current encoder layer). K i V i Q i Let be the i-th key matrix, i-th value matrix, and i-th query matrix obtained from the attention sublayer in the i-th encoder layer.

[0045] The self-attention output of the i-th encoder layer can be calculated using the following formula:

[0046] Among them, K i Vi Q i The attention sublayer of the i-th encoder layer contains the i-th key matrix, i-th value matrix, and i-th query matrix. The scaling factor is used to calculate the result. This is the self-attention output of the i-th encoder layer.

[0047] It is understandable that the M-layer encoder layers have the same structure, but different encoder layers use different first weight matrices, second weight matrices, and third weight matrices. The first, second, and third weight matrices of each encoder layer are all learned.

[0048] In the second embodiment, the M-1 encoder layers (excluding the first encoder layer) sequentially perform fusion processing on the fusion vector representation output by the previous encoder layer to obtain an enhanced vector representation. This may include: each encoder layer in the M-1 encoder layer performs encoding based on the fusion vector representation output by the previous encoder layer and the second vector representation corresponding to the second bird's-eye view based on a cross-attention mechanism to obtain the fusion vector representation output by the current encoder layer. This process continues until the last encoder layer in the M-1 encoder layer outputs the fusion vector representation, and the fusion vector representation output by the last encoder layer is used as the enhanced vector representation.

[0049] In the third implementation, the M-1 encoder layers (excluding the first encoder layer) in the M-1 encoder layer undergo fusion processing on the fusion vector representation output by the encoder layer above them to obtain an enhanced vector representation. This can include: the first N encoder layers in the M-1 encoder layer performing encoding based on the fusion vector representation output by the encoder layer above them and the second vector representation corresponding to the second bird's-eye view, to obtain the fusion vector representation output by the current encoder layer; the remaining encoder layers in the M-1 encoder layer (excluding the first N encoder layers) performing encoding based on the fusion vector representation output by the encoder layer above them and the self-attention mechanism, to obtain the fusion vector representation output by the current encoder layer, and so on, until the last encoder layer in the M-1 encoder layer outputs the fusion vector representation, which is then used as the enhanced vector representation.

[0050] It should be noted that in the third embodiment, each of the remaining encoder layers performs encoding based on the fusion vector representation output by the previous encoder layer using a self-attention mechanism. The relevant description of the fusion vector representation obtained by each encoder layer in the first embodiment can be referred to, and will not be repeated here for the sake of brevity.

[0051] It should be noted that in the second and third embodiments, the remaining encoder layers perform encoding based on the fused vector representation output by the previous encoder layer and the second vector representation corresponding to the second bird's-eye view using a cross-attention mechanism. This can include the following steps: The i-th encoder layer uses the first and second weight matrices of the i-th encoder layer to perform a linear transformation on the fused vector representation output by the (i-1)-th encoder layer, obtaining the i-th key matrix and the i-th value matrix; the i-th encoder layer uses the third weight matrix of the i-th encoder layer to perform a linear transformation on the second vector representation corresponding to the second bird's-eye view, obtaining the i-th query matrix; the i-th encoder layer performs self-attention interaction based on the i-th key matrix, the i-th value matrix, and the i-th query matrix to obtain the fused vector representation output by the i-th encoder layer. The specific calculation methods for obtaining the i-th key matrix, the i-th value matrix, and the i-th query matrix are as follows: K i =X bev_i-1 ×W Ki V i =X bev_i-1 ×W Vi Q i =F img_bev ×W Qi Among them, X bev_i-1 W represents the fused vector output of the (i-1)th encoder layer. Ki W Vi W Qi These correspond to the first, second, and third weight matrices in the i-th encoder layer, K. i V i Q i Let F be the i-th key matrix, the i-th value matrix, and the i-th query matrix. img_bev This is the second vector representation.

[0052] In some embodiments, the decoder makes predictions based on the augmented vector representation to obtain the target detection result of the target scene, which may include the following steps performed by the decoder: filtering out potential target regions in the second bird's-eye view; mapping the potential target regions to three-dimensional candidate centers; using the three-dimensional candidate centers as query vectors, the decoder performs regression prediction on the target region in the augmented vector representation and outputs the target detection result.

[0053] Understandably, the second input layer of the decoder preprocesses the input second bird's-eye view to filter out potential target regions in the second bird's-eye view. In some embodiments, the step of filtering out potential target regions in the second bird's-eye view may include: calculating the response intensity of each grid cell in the second bird's-eye view; selecting each grid cell in the second bird's-eye view with a response intensity greater than a preset intensity threshold as a potential target region, or filtering out the K grid cells with the highest response intensity in the second bird's-eye view as potential target regions, where K is an integer greater than 1.

[0054] In some embodiments, the response intensity of each grid cell is calculated using the channel mean or L2 norm. It can be understood that the potential target region is the high-response location in the second bird's-eye view. A Top-k strategy is used to select high-response locations, that is, selecting the top k grid cells with response intensities sorted from largest to smallest, resulting in a candidate location set. Alternatively, all grid cells with response intensities greater than a preset intensity threshold are selected to obtain the candidate location set. The candidate location set is the potential target region, and each candidate location in the potential target region is a high-response location.

[0055] It is understandable that mapping the potential target area to a three-dimensional candidate center means back-projecting the coordinates of each candidate position to the three-dimensional space of the vehicle coordinate system to obtain the three-dimensional candidate center corresponding to that candidate position, thereby obtaining each three-dimensional candidate center that corresponds one-to-one with each candidate position.

[0056] In some embodiments, the decoder uses the three-dimensional candidate center as the query vector, performs regression prediction on the target region in the enhanced vector representation, and outputs the target detection result, which may include the following steps 1-3: Step 1: The first decoder layer generates feature vectors for each candidate location in the potential target region. Based on the feature vectors of each candidate location, the query matrix of the decoder is obtained. The decoder performs a linear transformation on the augmented vector representation output by the encoder to obtain the key matrix and value matrix of the decoder. Based on the query matrix, key matrix, and value matrix, cross-attention interaction is performed to obtain the target vector representation.

[0057] Step 2: In the N-1 decoder layers (excluding the first decoder layer), the target vector representation output by the previous decoder layer and the augmented vector representation output by the encoder are cross-attentioned sequentially to obtain the next target vector representation. This process continues until the last decoder layer outputs the target vector representation.

[0058] Step 3: The output layer transforms the target vector representation output by the last decoder layer to obtain the target detection result.

[0059] Based on the same inventive concept, embodiments of the present invention provide a three-dimensional target detection device. Figure 2 A schematic diagram of the structure of a three-dimensional target detection device according to some embodiments of the present invention is shown. For example... Figure 2 As shown, the 3D target detection device includes: a data acquisition unit 201, used to acquire scene data of the target scene, the scene data being first point cloud data collected by LiDAR and multi-view images collected by a camera; an image generation unit 201, used to generate a first bird's-eye view based on the first point cloud data, and a second bird's-eye view based on the multi-view images; an input unit 201, used to input the first bird's-eye view and the second bird's-eye view into an attention-based encoder-decoder network, the encoder-decoder network including an encoder and a decoder; and an encoder-decoder unit 201, used to guide the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation, and the decoder to make predictions based on the enhanced vector representation to obtain the target detection result of the target scene.

[0060] In some embodiments, the image generation unit 201 includes: a first generation subunit, configured to perform voxel-level feature encoding on the first point cloud data through a first feature extraction network to obtain a first bird's-eye view; and to perform feature extraction on the multi-view image through a second feature extraction network to obtain corresponding multi-view image features; and a second generation subunit, configured to map the multi-view image features to the bird's-eye view space using the intrinsic and extrinsic parameter matrices of the camera to obtain a second bird's-eye view.

[0061] In some embodiments, the first feature extraction network is a three-dimensional convolutional neural network based on voxel networks, and the second feature extraction network is a two-dimensional convolutional neural network.

[0062] In some embodiments, the second generation subunit is configured to: convert the pixels in the multi-view image features into three-dimensional coordinate points in the camera coordinate system using the intrinsic and extrinsic parameter matrices of the camera; convert the three-dimensional coordinate points to the vehicle coordinate system using the extrinsic parameter matrix to obtain second point cloud data; and project the second point cloud data onto the bird's-eye view space to obtain the second bird's-eye view.

[0063] In some embodiments, the encoder includes a first input layer and an M-layer encoder layer, where M is an integer of 1. The encoding / decoding unit 201 includes an encoding subunit, configured to: convert the first bird's-eye view into a first vector representation and convert the second bird's-eye view into a second vector representation through the first encoder layer in the M-layer encoder layer; perform a linear transformation on the first vector representation using a first weight matrix to obtain a first key matrix, perform a linear transformation on the first vector representation using a second weight matrix to obtain a first value matrix, and perform a linear transformation on the second vector representation using a third weight matrix to obtain a first query matrix; the first encoder layer performs cross-attention interaction based on the first key matrix, the first value matrix, and the first query matrix to obtain a fused vector representation; and the M-1 encoder layers in the M-layer encoder layer, excluding the first encoder layer, sequentially perform fusion processing on the fused vector representation output by the previous encoder layer to obtain the enhanced vector representation.

[0064] In some embodiments, the encoding / decoding unit 201 includes a decoding subunit, configured to: filter out potential target regions in the second bird's-eye view; map the potential target regions to three-dimensional candidate centers; and use the three-dimensional candidate centers as query vectors to perform regression prediction on the target region in the enhanced vector representation, and output the target detection result.

[0065] In some embodiments, the decoding subunit is configured to: calculate the response intensity of each grid cell in the second bird's-eye view; Alternatively, select from the second bird's-eye view each grid cell whose response intensity is greater than a preset intensity threshold as the potential target area, or select from the second bird's-eye view the K grid cells with the highest response intensity as the potential target area.

[0066] In this embodiment, the three-dimensional target detection device is the device for implementing the aforementioned three-dimensional target detection method embodiment. More implementation details of the three-dimensional target detection device can be found in the description of the aforementioned three-dimensional target detection method embodiment. For the sake of brevity, these details will not be repeated here.

[0067] Based on the same inventive concept, this invention provides an electronic device, which is a vehicle infotainment system in the field of autonomous driving. Figure 3 A schematic diagram of an electronic device according to some embodiments of the present invention is shown. (Reference) Figure 3 As shown, the electronic device includes a memory 304, a processor 302, and a computer program stored in the memory 304 and executable on the processor 302. When the processor 302 executes the computer program, it implements the three-dimensional target detection method provided in any embodiment of the present invention.

[0068] Among them, Figure 3 In this document, a bus architecture (represented by bus 300) is used. Bus 300 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 can be used to store data used by processor 302 during operation.

[0069] Based on the same inventive concept, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the three-dimensional target detection method provided in any embodiment of the present invention.

[0070] The three-dimensional target detection method, electronic device, and medium provided in the embodiments of the present invention achieve at least the following technical effects or advantages: Early fusion and late fusion, with their simple modal stitching, either fuse on the image plane or crudely project or flatten image features. Image features are unaware of their corresponding location in the bird's-eye view generated from the point cloud data, resulting in spatial misalignment and a lack of stable 3D spatial anchors for image semantics. Fusion relies on the model's guesswork. This invention, however, rigorously maps multi-view image features to the BEV space at the feature level, making the image features themselves BEV features with 3D geometric meaning. In the Transformer structure, the feature representations of the point cloud data and the image feature representation are fused for guided queries, fully extracting the joint information of image semantics and geometry. This allows image semantics to directly influence the modeling focus of point cloud features during the Transformer encoding stage, significantly improving the point cloud representation capability of target regions. The image-guided attention mechanism can compensate for sparse areas of the point cloud at a distance, improving the detection accuracy of distant or occluded targets, thus enhancing long-range robustness. The high-efficiency voxel backbone structure based on CenterPoint has controllable overall computational overhead and can maintain real-time performance, making it suitable for real-time scenarios such as autonomous driving.

[0071] In related technologies, the query matrix of the decoder in the Transformer architecture typically uses learnable queries or uniform sampling of BEV grids, resulting in the query vector itself not knowing where the target is. This leads to a large amount of invalid computation in the decoding stage, with the decoder "blindly searching" for the target. This invention, however, seems to attempt a decoding method that generates 3D candidate centers based on the response of BEV features from multi-view images. This allows the target prediction process to be driven by both image semantics and 3D geometric information, performing fine regression only around these potential target regions, reducing invalid queries and improving detection accuracy.

[0072] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0074] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0076] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A three-dimensional target detection method, characterized in that, include: Acquire scene data of the target scene, the scene data including first point cloud data collected by lidar at the same time and multi-view images collected by camera; A first bird's-eye view is generated based on the first point cloud data, and a second bird's-eye view is generated based on the multi-view images; The first bird's-eye view and the second bird's-eye view are input into an attention-based encoding and decoding network, which includes an encoder and a decoder. The encoder is guided by the second bird's-eye view to model the first bird's-eye view to obtain an enhanced vector representation. The decoder then makes predictions based on the enhanced vector representation to obtain the target detection result of the target scene.

2. The three-dimensional target detection method as described in claim 1, characterized in that, The steps of generating a first bird's-eye view based on the first point cloud data and generating a second bird's-eye view based on the multi-view images include: The first feature extraction network performs voxel-level feature encoding on the first point cloud data to obtain the first bird's-eye view; The second feature extraction network extracts features from the multi-view images to obtain the corresponding multi-view image features. The multi-view image features are mapped to the bird's-eye view space using the intrinsic and extrinsic parameter matrices of the camera to obtain the second bird's-eye view.

3. The three-dimensional target detection method as described in claim 2, characterized in that, The first feature extraction network is a three-dimensional convolutional neural network based on voxel networks, and the second feature extraction network is a two-dimensional convolutional neural network.

4. The three-dimensional target detection method as described in claim 2, characterized in that, The step of mapping the multi-view image features to a bird's-eye view space using the intrinsic and extrinsic parameter matrices of the camera to obtain the second bird's-eye view includes: Using the intrinsic and extrinsic parameter matrices of the camera, the pixels in the multi-view image features are converted into three-dimensional coordinates in the camera coordinate system. The extrinsic parameter matrix is ​​then used to convert the three-dimensional coordinates to the vehicle coordinate system to obtain the second point cloud data. The second point cloud data is projected onto the bird's-eye view space to obtain the second bird's-eye view.

5. The three-dimensional target detection method as described in claim 1, characterized in that, The encoder includes a first input layer and M encoder layers, where M is an integer of 1. The step of guiding the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation includes: The first input layer converts the first bird's-eye view into a first vector representation and the second bird's-eye view into a second vector representation; The first encoder layer in the M-layer encoder layer performs a linear transformation on the first vector representation using a first weight matrix to obtain a first key matrix, performs a linear transformation on the first vector representation using a second weight matrix to obtain a first value matrix, and performs a linear transformation on the second vector representation using a third weight matrix to obtain a first query matrix. The first encoder layer performs cross-attention interaction based on the first key matrix, the first value matrix, and the first query matrix to obtain a fused vector representation; In the M-layer encoder layer, except for the first layer encoder layer, the M-1 layer encoder layers sequentially perform fusion processing on the fusion vector representation output by the previous layer encoder layer to obtain the enhanced vector representation.

6. The three-dimensional target detection method as described in claim 1, characterized in that, The decoder makes predictions based on the enhanced vector representation to obtain the target detection results for the target scene, including: Filter out potential target areas in the second bird's-eye view; Map the potential target region as a three-dimensional candidate center; The decoder uses the three-dimensional candidate center as the query vector, performs regression prediction on the target region in the enhanced vector representation, and outputs the target detection result.

7. The three-dimensional target detection method as described in claim 6, characterized in that, The process of filtering out potential target areas in the second bird's-eye view includes: Calculate the response intensity of each grid cell in the second bird's-eye view; Alternatively, select from the second bird's-eye view each grid cell whose response intensity is greater than a preset intensity threshold as the potential target area, or select from the second bird's-eye view the K grid cells with the highest response intensity as the potential target area.

8. A three-dimensional target detection device, characterized in that, include: The data acquisition unit is used to acquire scene data of the target scene, which includes first point cloud data collected by lidar and multi-view images collected by camera. The image generation unit is used to generate a first bird's-eye view based on the first point cloud data, and to generate a second bird's-eye view based on the multi-view image; An input unit is used to input the first bird's-eye view and the second bird's-eye view into an attention-based encoding and decoding network, the encoding and decoding network including an encoder and a decoder; The encoding / decoding unit is used to guide the encoder to model the first bird's-eye view based on the second bird's-eye view to obtain an enhanced vector representation, and the decoder makes predictions based on the enhanced vector representation to obtain the target detection result of the target scene.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional target detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the three-dimensional target detection method according to any one of claims 1-7.

Images