Target detection method, and three-dimensional target detection model training method and device

Abstract

The embodiment of the invention provides a target detection method and a three-dimensional target detection model training method and device, and relates to the technical field of machine vision. The target detection method comprises the following steps: acquiring a to-be-detected point cloud and a to-be-detected image of a to-be-detected space region; extracting to-be-detected initial point cloud features, and obtaining to-be-detected initial image features; based on the current coordinate and the reference feature of each reference three-dimensional point, obtaining the current to-be-detected fusion point cloud feature of the reference three-dimensional point; obtaining a current to-be-detected intermediate image feature of the reference three-dimensional point based on the to-be-detected initial image feature; updating the current coordinate and reference feature of the reference three-dimensional point based on the current to-be-detected fusion point cloud feature and the to-be-detected intermediate image feature; returning to execute the step of extracting the features of the initial point cloud to be detected until the preset number of decoding times is reached; and obtaining the position of the object in the to-be-detected space region based on the current coordinate of each reference three-dimensional point. The accuracy of the obtained target detection result can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a target detection method, a three-dimensional target detection model training method and a device. Background Art

[0002] In the field of machine vision technology, visual sensors can be used to capture images of a spatial area to be detected, and target detection can be performed on the captured images to obtain detection results of the positions of objects in the spatial area to be detected.

[0003] However, the process of image acquisition by visual sensors is susceptible to external interference such as changes in lighting, background color, and occlusions, which can lead to low accuracy of target detection results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a target detection method, a three-dimensional target detection model training method and a device to improve the accuracy of the target detection results obtained. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a target detection method is first provided, the method comprising: obtaining a point cloud and an image of a spatial area to be detected, as the point cloud to be detected and the image to be detected, respectively; dividing the point cloud to be detected according to a bird's-eye view BEV grid, and extracting the initial point cloud features to be detected of each BEV grid, and performing feature extraction on the image to be detected to obtain the initial image features to be detected; for each reference 3D point, based on the current coordinates and reference features of the reference 3D point, sampling the BEV grids within the neighborhood of the reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grids to obtain the current fused point cloud features to be detected of the reference 3D point; obtaining the corresponding part of the reference 3D point in the initial image features to be detected, and obtaining the reference 3D point. 3D point, and the like.

[0006] Optionally, the step of dividing the point cloud to be detected according to the bird's-eye view BEV grids, extracting the features of the initial point cloud to be detected of each BEV grid, and performing feature extraction on the image to be detected to obtain features of the initial image to be detected includes: dividing the point cloud to be detected according to the bird's-eye view BEV grids, and determining the three-dimensional points to be detected contained in each BEV grid obtained by the division; using the point cloud feature extraction network in the pre-trained three-dimensional target detection model to perform feature extraction on the information to be used of the three-dimensional points to be detected contained in each BEV grid to obtain features of the initial point cloud to be detected of each BEV grid; and inputting the image to be detected into the three-dimensional target detection model. The image feature extraction network is used to obtain multiple initial image features to be detected; wherein the information to be used of a 3D point includes: the coordinates of the 3D point and the coordinates of the centroid of the BEV grid to which the 3D point belongs; the 3D target detection model is obtained by training based on the sample image, the sample point cloud and the sample label representing the position of the object in the sample space area; the 3D target detection model also includes a deformable attention network, a weighted attention network and a decoder; for each reference 3D point, based on the current coordinates and reference features of the reference 3D point, the BEV grid in the neighborhood of the reference 3D point is sampled, and the sampled BEV grid is sampled. The initial point cloud features are detected and fused to obtain the current fused point cloud features to be detected of the reference 3D point, including: for each reference 3D point, the current reference features of the reference 3D point are input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; the initial point cloud features to be detected of the BEV grids corresponding to each position offset are fused according to the weights corresponding to each position offset to obtain the current fused point cloud features to be detected of the reference 3D point The method comprises: measuring fused point cloud features; obtaining the corresponding part of the reference 3D point in the initial image features to be detected to obtain the current intermediate image features to be detected of the reference 3D point, including: inputting the current reference features of the reference 3D point into the weighted attention network to obtain the weight of each initial image feature to be detected; fusing the corresponding feature values ​​of the reference 3D point in each initial image feature to be detected according to the weight of each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference 3D point; decoding the current splicing features to be detected to obtain the current decoding result, including: inputting the current splicing features to be detected into the decoder to obtain the current decoding result.

[0007] Optionally, the three-dimensional target detection model also includes a multi-layer perceptron; before the current reference feature of each reference three-dimensional point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset, the method also includes: inputting the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is: calibrated according to the acquisition method of the point cloud to be detected and the image to be detected; using the obtained correction amount The initial transformation matrix is ​​corrected; and the eigenvalues ​​corresponding to the reference three-dimensional point in each of the initial image features to be detected are fused according to the weights of the initial image features to be detected to obtain the current intermediate image features to be detected of the reference three-dimensional point, including: using the corrected transformation matrix to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the image to be detected as the reference coordinates to be detected; according to the weights of the initial image features to be detected, the eigenvalues ​​corresponding to the reference coordinates to be detected in each of the initial image features to be detected are fused to obtain the current intermediate image features to be detected of the reference three-dimensional point.

[0008] Optionally, the initial point cloud features to be detected of each BEV grid and the initial image features to be detected are input into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix, including: inputting the initial point cloud features to be detected of each BEV grid, the initial image features to be detected, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix.

[0009] Optionally, before fusing the eigenvalues ​​corresponding to the reference coordinates to be detected in each initial image feature to be detected according to the weights of each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference three-dimensional point, the method also includes: if the reference coordinate to be detected is not an integer, obtaining other pixel coordinates within the neighborhood of the reference coordinate to be detected; and for each initial image feature to be detected, interpolating the eigenvalues ​​corresponding to the other obtained pixel coordinates in the initial image feature to be detected as the eigenvalues ​​corresponding to the reference coordinate to be detected in the initial image feature to be detected.

[0010] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the image to be detected is one, and the multiple initial image features to be detected include: image features of the image to be detected extracted by the multiple downsampling layers; or, the image to be detected includes images of the spatial area to be detected captured at multiple different shooting angles, and the multiple initial image features to be detected include image features of each image to be detected.

[0011] Optionally, the determining of the three-dimensional points to be detected contained in each BEV grid obtained by division includes: for each BEV grid obtained by division, if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than a specified number, sampling the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is equal to the specified number, using the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, calculating the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid, and generating three-dimensional points whose coordinates are preset values ​​of the difference; combining the generated three-dimensional points and the three-dimensional points in the point cloud to be detected contained in the BEV grid, the three-dimensional points to be detected contained in the BEV grid are obtained.

[0012] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the detected real position corresponding to the three-dimensional point.

[0013] In a second aspect of an embodiment of the present application, a three-dimensional target detection model training method is provided, the method comprising: obtaining a point cloud and an image of a sample space area as a sample point cloud and a sample image, respectively, and obtaining a sample label containing the position of an object in the sample space area; dividing the sample point cloud according to a bird's-eye view BEV grid, and determining the sample three-dimensional points contained in each BEV grid obtained by the division; using a point cloud feature extraction network in a three-dimensional target detection model of an initial structure, extracting features from the sample three-dimensional points contained in each BEV grid to obtain sample initial point cloud features of each BEV grid; and inputting the sample image into the image feature extraction network in the three-dimensional target detection model. Extraction network, obtain multiple sample initial image features; wherein, the information to be used of a 3D point includes: the coordinates of the 3D point, and the coordinates of the centroid of the BEV grid to which the 3D point belongs; the 3D target detection model also includes a deformable attention network, a weighted attention network and a decoder; for each reference 3D point, the current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; according to each position The weight corresponding to the offset is used to fuse the sample initial point cloud features of the BEV grid corresponding to each position offset to obtain the current sample fused point cloud features of the reference 3D point; the current reference features of the reference 3D point are input into the weighted attention network to obtain the weight of each sample initial image feature; according to the weight of each sample initial image feature, the feature values ​​corresponding to the reference 3D point in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference 3D point; the current sample fused point cloud features and sample intermediate image features of each reference 3D point are spliced ​​to obtain the current sample splicing features; the current sample splicing features are input into the decoder to obtain the current decoding result, and based on The current decoding result updates the current coordinates and reference features of the reference 3D point; the step of inputting the current reference features of the reference 3D point into the deformable attention network for each reference 3D point to obtain a preset number of sampling position offsets and a weight corresponding to each position offset is returned to be executed until the number of decoding times reaches a preset number; based on the current coordinates of each reference 3D point, a detection result is obtained; wherein the detection result includes the position of the object in the sample space area; based on the difference between the sample label and the detection result, the model parameters of the 3D target detection model of the initial structure are adjusted until a preset convergence condition is reached to obtain a trained 3D target detection model.

[0014] Optionally, the three-dimensional target detection model also includes a multi-layer perceptron; before the current reference feature of each reference three-dimensional point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset, the method also includes: inputting the sample initial point cloud features of each BEV grid and the sample initial image features into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is ​​obtained by calibration according to the acquisition method of the sample point cloud and the sample image; the initial transformation matrix is ​​corrected using the obtained correction amount; according to the weight of each sample initial image feature, the eigenvalues ​​corresponding to the reference three-dimensional point in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference three-dimensional point, including: using the corrected transformation matrix to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the sample image as the sample reference coordinates; according to the weight of each sample initial image feature, the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference three-dimensional point.

[0015] Optionally, the sample initial point cloud features of each BEV grid and the sample initial image features are input into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix, including: inputting the sample initial point cloud features of each BEV grid, the sample initial image features, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix.

[0016] Optionally, before fusing the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature according to the weights of each sample initial image feature to obtain the current sample intermediate image feature of the reference three-dimensional point, the method also includes: if the sample reference coordinate is not an integer, obtaining other pixel coordinates within the neighborhood of the sample reference coordinate; and for each sample initial image feature, interpolating the eigenvalues ​​corresponding to the other obtained pixel coordinates in the sample initial image feature as the eigenvalue corresponding to the sample reference coordinate in the sample initial image feature.

[0017] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the sample image is one, and the multiple sample initial image features include: image features of the sample image extracted by the multiple downsampling layers; or, the sample image includes images of the sample space area captured at multiple different shooting angles, and the multiple sample initial image features include image features of each sample image.

[0018] Optionally, the determining of the sample three-dimensional points contained in each BEV grid obtained by division includes: for each BEV grid obtained by division, if the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than a specified number, sampling the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is equal to the specified number, using the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, calculating the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid, and generating three-dimensional points whose coordinates are preset values ​​of the difference; combining the generated three-dimensional points with the three-dimensional points in the sample point cloud contained in the BEV grid, the sample three-dimensional points contained in the BEV grid are obtained.

[0019] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the real position corresponding to the detected three-dimensional point.

[0020] Optionally, the sample label also includes the category of the object in the sample space area; the position indication of the object in the sample space area includes: the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object; the detection result also includes the category of the object in the sample space area; the difference between the sample label and the detection result includes: the difference in the position of the object in the sample space area, and the difference in the category of the object in the sample space area.

[0021] In a third aspect of an embodiment of the present application, a target detection device is provided, the device comprising: a first acquisition module, used to acquire a point cloud and an image of a spatial area to be detected, as the point cloud to be detected and the image to be detected, respectively; a first feature extraction module, used to divide the point cloud to be detected according to a bird's-eye view BEV grid, and extract the initial point cloud features to be detected of each BEV grid, and perform feature extraction on the image to be detected to obtain the initial image features to be detected; a first point cloud feature fusion module, used to sample the BEV grids within the neighborhood of the reference three-dimensional point for each reference three-dimensional point based on the current coordinates and reference features of the reference three-dimensional point, and fuse the initial point cloud features to be detected of the sampled BEV grids to obtain the current fused point cloud features to be detected of the reference three-dimensional point; a first image feature fusion module, used to obtain the corresponding coordinates of the reference three-dimensional point in the initial image features to be detected part, obtaining the current intermediate image features to be detected of the reference 3D point; a first splicing module, used to splice the current fused point cloud features to be detected and the intermediate image features to be detected of each reference 3D point, to obtain the current splicing features to be detected; a first updating module, used to decode the current splicing features to be detected, to obtain the current decoding result, and to update the current coordinates and reference features of the reference 3D point based on the current decoding result; returning to execute the steps of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point for each reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches a preset number; a position acquisition module, used to obtain the position of the object in the spatial area to be detected based on the current coordinates of each reference 3D point.

[0022] Optionally, the first feature extraction module is specifically used to divide the point cloud to be detected according to the bird's-eye view BEV grid, and determine the three-dimensional points to be detected contained in each BEV grid obtained by the division; use the point cloud feature extraction network in the pre-trained three-dimensional target detection model to extract features of the to-be-detected three-dimensional points contained in each BEV grid to obtain the initial point cloud features to be detected of each BEV grid; and input the image to be detected into the image feature extraction network in the three-dimensional target detection model to obtain multiple initial image features to be detected; wherein the to-be-detected information of a three-dimensional point includes: the coordinates of the three-dimensional point, and the coordinates of the center of mass of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model is: trained based on sample images, sample point clouds and sample labels representing the positions of objects in the sample space area; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder; the first point cloud feature fusion module is specifically used to extract features of the reference three-dimensional point for each reference three-dimensional point. The current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; the initial point cloud features to be detected of the BEV grids corresponding to each position offset are fused according to the weights corresponding to each position offset to obtain the current fused point cloud features to be detected of the reference 3D point; the first image feature fusion module is specifically used to input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each initial image feature to be detected; according to the weights of each initial image feature to be detected, the feature values ​​corresponding to the reference 3D point in each initial image feature to be detected are fused to obtain the current intermediate image feature to be detected of the reference 3D point; the first update module is specifically used to input the current splicing feature to be detected into the decoder to obtain the current decoding result.

[0023] Optionally, the three-dimensional target detection model also includes a multi-layer perceptron; the device also includes: a first correction amount acquisition module, which is used to input the current reference feature of the reference three-dimensional point into the deformable attention network before the first point cloud feature fusion module executes the input of the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix for each reference three-dimensional point, and obtains the correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is: calibrated according to the acquisition method of the point cloud to be detected and the image to be detected; the initial transformation matrix is ​​corrected using the obtained correction amount; the first image feature fusion module is specifically used to use the corrected transformation matrix to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the image to be detected as the reference coordinates to be detected; according to the weights of each initial image feature to be detected, the eigenvalues ​​corresponding to the reference coordinates to be detected in each initial image feature to be detected are fused to obtain the current intermediate image feature to be detected of the reference three-dimensional point.

[0024] Optionally, the first correction amount acquisition module is specifically used to input the initial point cloud features to be detected of each BEV grid, the initial image features to be detected, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0025] Optionally, the device also includes: a first interpolation module, which is used to execute, in the first image feature fusion module, fusing the eigenvalues ​​corresponding to the reference coordinate to be detected in each initial image feature to be detected according to the weights of each initial image feature to be detected, and before obtaining the current intermediate image feature to be detected of the reference three-dimensional point, if the reference coordinate to be detected is not an integer, obtaining other pixel coordinates within the neighborhood of the reference coordinate to be detected; for each initial image feature to be detected, interpolating the eigenvalues ​​corresponding to the other acquired pixel coordinates in the initial image feature to be detected as the eigenvalues ​​corresponding to the reference coordinate to be detected in the initial image feature to be detected.

[0026] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the image to be detected is one, and the multiple initial image features to be detected include: image features of the image to be detected extracted by the multiple downsampling layers; or, the image to be detected includes images of the spatial area to be detected captured at multiple different shooting angles, and the multiple initial image features to be detected include image features of each image to be detected.

[0027] Optionally, the first feature extraction module is specifically used to, for each BEV grid obtained by division, if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than a specified number, sample the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is equal to the specified number, use the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, calculate the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid, and generate three-dimensional points whose coordinates are preset values ​​of the difference; combine the generated three-dimensional points and the three-dimensional points in the point cloud to be detected contained in the BEV grid to obtain the three-dimensional points to be detected contained in the BEV grid.

[0028] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the detected real position corresponding to the three-dimensional point.

[0029] In a fourth aspect of an embodiment of the present application, a three-dimensional object detection model training device is provided, the device comprising:

[0030] The second acquisition module is used to acquire a point cloud and an image of a sample space region as a sample point cloud and a sample image, respectively, and to acquire a sample label containing the position of an object in the sample space region; the second feature extraction module is used to divide the sample point cloud according to the bird's-eye view BEV grid, and determine the sample three-dimensional points contained in each BEV grid obtained by the division; use the point cloud feature extraction network in the three-dimensional target detection model of the initial structure to extract features from the sample three-dimensional points to be used contained in each BEV grid, and obtain the sample initial point cloud features of each BEV grid; and input the sample image into the image feature extraction network in the three-dimensional target detection model to obtain multiple Sample initial image features; wherein the information to be used of a 3D point includes: the coordinates of the 3D point and the coordinates of the centroid of the BEV grid to which the 3D point belongs; the 3D target detection model also includes a deformable attention network, a weighted attention network and a decoder; a second point cloud feature fusion module is used to, for each reference 3D point, input the current reference feature of the reference 3D point into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, use the position offset to offset the current coordinates of the reference 3D point, and determine the BEV grid to which the offset coordinates belong as the position offset for The BEV grid corresponding to each position offset is fused according to the weight corresponding to each position offset, so as to obtain the current sample fused point cloud feature of the reference 3D point; the second image feature fusion module is used to input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each sample initial image feature; according to the weight of each sample initial image feature, the feature value corresponding to the reference 3D point in each sample initial image feature is fused to obtain the current sample intermediate image feature of the reference 3D point; the second splicing module is used to splice the current sample fused point cloud feature and the sample intermediate image feature of each reference 3D point , obtain the current sample splicing feature; a second updating module, used to input the current sample splicing feature into the decoder, obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the step of inputting the current reference feature of the reference 3D point into the deformable attention network for each reference 3D point, obtaining a preset number of sampling position offsets, and a weight corresponding to each position offset, until the number of decoding times reaches the preset number; a detection result acquisition module, used to obtain a detection result based on the current coordinates of each reference 3D point; wherein the detection result includes the position of the object in the sample space area;A training module is used to adjust the model parameters of the three-dimensional object detection model of the initial structure based on the difference between the sample label and the detection result until a preset convergence condition is reached to obtain a trained three-dimensional object detection model. ;

[0031] Optionally, the three-dimensional target detection model also includes a multi-layer perceptron; the device also includes: a second correction amount acquisition module, which is used to input the sample initial point cloud features of each BEV grid and the sample initial image features into the multi-layer perceptron before the second point cloud feature fusion module executes the current reference features of each reference three-dimensional point for each reference three-dimensional point, and obtains a preset number of sampling position offsets and a weight corresponding to each position offset, and obtains a correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is: calibrated according to the acquisition method of the sample point cloud and the sample image; the initial transformation matrix is ​​corrected using the obtained correction amount; the second image feature fusion module is specifically used to: use the corrected transformation matrix to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the sample image as the sample reference coordinates; according to the weight of each sample initial image feature, the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature are fused to obtain the current sample intermediate image features of the reference three-dimensional point.

[0032] Optionally, the second correction amount acquisition module is specifically used to input the sample initial point cloud features of each BEV grid, the sample initial image features, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0033] Optionally, the device also includes: a second interpolation module, which is used to execute, before the second image feature fusion module executes the fusion of the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature according to the weights of each sample initial image feature, to obtain the current sample intermediate image feature of the reference three-dimensional point, if the sample reference coordinate is not an integer, then obtain other pixel coordinates within the neighborhood of the sample reference coordinate; for each sample initial image feature, interpolate the eigenvalues ​​corresponding to the other obtained pixel coordinates in the sample initial image feature as the eigenvalue corresponding to the sample reference coordinate in the sample initial image feature.

[0034] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the sample image is one, and the multiple sample initial image features include: image features of the sample image extracted by the multiple downsampling layers; or, the sample image includes images of the sample space area captured at multiple different shooting angles, and the multiple sample initial image features include image features of each sample image.

[0035] Optionally, the second feature extraction module is specifically used to, for each BEV grid obtained by division, if the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than a specified number, sample the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is equal to the specified number, use the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, calculate the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid, and generate three-dimensional points whose coordinates are preset values; combine the generated three-dimensional points and the three-dimensional points in the sample point cloud contained in the BEV grid to obtain the sample three-dimensional points contained in the BEV grid.

[0036] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the real position corresponding to the detected three-dimensional point.

[0037] Optionally, the sample label also includes the category of the object in the sample space area; the position indication of the object in the sample space area includes: the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object; the detection result also includes the category of the object in the sample space area; the difference between the sample label and the detection result includes: the difference in the position of the object in the sample space area, and the difference in the category of the object in the sample space area.

[0038] In the fifth aspect of the embodiments of the present application, an electronic device is provided, comprising: a memory for storing computer programs; a processor for implementing the target detection method described in any one of the first aspects above, or implementing the three-dimensional target detection model training method described in any one of the second aspects above, when executing the program stored in the memory.

[0039] In a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the target detection method described in any one of the first aspects above is implemented, or the three-dimensional target detection model training method described in any one of the second aspects above is implemented.

[0040] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any target detection method described in the first aspect above, or execute any three-dimensional target detection model training method described in the second aspect above.

[0041] An embodiment of the present application provides a target detection method, in which an electronic device obtains a point cloud and an image of a spatial area to be detected, respectively as a point cloud to be detected and an image to be detected; the point cloud to be detected is divided according to BEV grids, and the initial point cloud features to be detected of each BEV grid are extracted, and feature extraction is performed on the image to be detected to obtain features of the initial image to be detected; for each reference 3D point, based on the current coordinates and reference features of the reference 3D point, the BEV grids within the neighborhood of the reference 3D point are sampled, and the initial point cloud features to be detected of the sampled BEV grids are fused to obtain the current fused point cloud features to be detected of the reference 3D point; the corresponding part of the reference 3D point in the initial image features to be detected is obtained, and the current fused point cloud features to be detected of the reference 3D point is obtained. intermediate image features; stitching the current fused point cloud features to be detected and the intermediate image features to be detected of each reference 3D point to obtain the current stitching features to be detected; decoding the current stitching features to be detected to obtain the current decoding result, and updating the current coordinates and reference features of the reference 3D point based on the current decoding result; returning to execute the steps of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point for each reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches the preset number; obtaining the position of the object in the spatial area to be detected based on the current coordinates of each reference 3D point.

[0042] Based on the above processing, the electronic device can obtain the current fused point cloud features to be detected based on the point cloud to be detected; and obtain the current intermediate image features to be detected based on the image to be detected. Based on the current fused point cloud features to be detected and the current intermediate image features to be detected, the current coordinates and reference features of each reference three-dimensional point are updated. After a preset number of updates, the position of the object in the spatial area to be detected is determined according to the current coordinates of each reference three-dimensional point. That is, target detection is performed on the spatial area to be detected based on the point cloud and the image. In this way, since the process of acquiring the point cloud is not easily affected by external interference such as changes in lighting, changes in background color, and obstructions, combining the point cloud and the image to perform target detection together can make up for the defect of low accuracy of the target detection results obtained based only on the image, reduce the impact of external interference on the accuracy of the target detection results, and improve the accuracy of the target detection results obtained.

[0043] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0045] Figure 1 A first flow chart of the target detection method provided in the embodiment of the present application;

[0046] Figure 2 A second flow chart of the target detection method provided in the embodiment of the present application;

[0047] Figure 3 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a learnable radar matrix mapping sampling module 302 provided in an embodiment of the present application;

[0049] Figure 5 A flowchart of a three-dimensional object detection model training method provided in an embodiment of the present application;

[0050] Figure 6 A structural diagram of a target detection device provided in an embodiment of the present application;

[0051] Figure 7 A structural diagram of a three-dimensional object detection model training device provided in an embodiment of the present application;

[0052] Figure 8 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0054] In the field of machine vision technology, visual sensors can be used to collect images of the space area to be detected, and the detection results of the position of the object in the space area to be detected can be obtained by performing target detection on the collected images. However, the process of collecting images by visual sensors is easily affected by external interference such as changes in lighting, changes in background color, and obstructions, which will lead to low accuracy of target detection results.

[0055] In order to solve the above problems, the present application provides a target detection method, which is applied to electronic devices. The electronic device can be a terminal or a server. The electronic device can obtain a point cloud to be detected and an image to be detected in a spatial area to be detected; divide the point cloud to be detected according to the BEV (Bird's Eye View) grid, and extract the initial point cloud features to be detected of each BEV grid, and extract features of the image to be detected to obtain the initial image features to be detected; for each reference three-dimensional point, based on the current coordinates and reference features of the reference three-dimensional point, sample the BEV grids within the neighborhood of the reference three-dimensional point, and fuse the initial point cloud features to be detected of the sampled BEV grids to obtain the current fused point cloud features to be detected of the reference three-dimensional point; obtain the corresponding part of the reference three-dimensional point in the initial image features to be detected, and obtain the current intermediate image features to be detected of the reference three-dimensional point; the current fused point cloud to be detected of each reference three-dimensional point The features and the features of the intermediate image to be detected are spliced ​​to obtain the current spliced ​​features to be detected; the current spliced ​​features to be detected are decoded to obtain the current decoding results, and the current coordinates and reference features of the reference 3D point are updated based on the current decoding results; the steps of sampling the BEV grid within the neighborhood of each reference 3D point based on the current coordinates and reference features of the reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point are returned to be executed until the number of decodings reaches the preset number; based on the current coordinates of each reference 3D point, the position of the object in the spatial area to be detected is obtained. The point cloud and image are combined for target detection to improve the accuracy of the target detection results.

[0056] See also Figure 1 , Figure 1 The first flow chart of the target detection method provided in the embodiment of the present application, the method may include the following steps:

[0057] S101: Acquire a point cloud and an image of a spatial area to be detected, as a point cloud to be detected and an image to be detected, respectively.

[0058] S102: Divide the point cloud to be detected according to the bird's-eye view grids, extract the initial point cloud features to be detected of each bird's-eye view grid, and perform feature extraction on the image to be detected to obtain the initial image features to be detected.

[0059] S103: For each reference 3D point, based on the current coordinates and reference features of the reference 3D point, the bird's-eye view grid within the neighborhood of the reference 3D point is sampled, and the initial point cloud features to be detected of the sampled bird's-eye view grid are fused to obtain the current fused point cloud features to be detected of the reference 3D point.

[0060] S104: Acquire the portion of the reference three-dimensional point corresponding to the initial image feature to be detected, and obtain the current intermediate image feature to be detected of the reference three-dimensional point.

[0061] S105: stitching the current fused point cloud features to be detected and the intermediate image features to be detected of each reference three-dimensional point to obtain the current stitched features to be detected.

[0062] S106: Decode the current splicing features to be detected to obtain the current decoding results, and update the current coordinates and reference features of the reference 3D point based on the current decoding results; return to execute the steps of sampling the bird's-eye view grid within the neighborhood of each reference 3D point based on the current coordinates and reference features of the reference 3D point, and fusing the sampled initial point cloud features to be detected of the bird's-eye view grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches the preset number.

[0063] S107: Based on the current coordinates of each reference three-dimensional point, the position of the object in the spatial area to be detected is obtained.

[0064] Based on the target detection method provided in the embodiment of the present application, the electronic device can obtain the current fused point cloud features to be detected based on the point cloud to be detected; and obtain the current intermediate image features to be detected based on the image to be detected. Based on the current fused point cloud features to be detected and the current intermediate image features to be detected, the current coordinates and reference features of each reference three-dimensional point are updated. After a preset number of updates, the position of the object in the spatial area to be detected is determined according to the current coordinates of each reference three-dimensional point. That is, target detection is performed on the spatial area to be detected based on the point cloud and the image. In this way, since the process of acquiring the point cloud is not easily affected by external interference such as changes in lighting, changes in background color, and obstructions, combining the point cloud and the image to perform target detection together can make up for the defect of low accuracy of the target detection results obtained based only on the image, reduce the impact of external interference on the accuracy of the target detection results, and improve the accuracy of the target detection results obtained.

[0065] For step S101, the spatial area to be detected and the types of objects that may exist therein are related to the business needs of the application scenario. For example, in a digital intersection scenario that integrates multiple sensors and intelligent technologies to achieve modern traffic management that fully perceives, analyzes, and controls traffic information at intersections, the spatial area to be detected may include roads around the intersection, and objects that may exist therein may include vehicles, roadblocks, etc. In an intelligent loading scenario, the spatial area to be detected may be the cargo placement area inside the carriage, and objects that may exist therein may include baffles, latches, air conditioner units, etc.

[0066] By scanning the spatial area to be detected by a visual sensor, an image of the spatial area to be detected can be collected; by scanning the spatial area to be detected by a radar sensor, a point cloud of the spatial area to be detected can be collected. The electronic device can obtain an image of the spatial area to be detected from the visual sensor as an image to be detected; and obtain a point cloud of the spatial area to be detected from the radar sensor as a point cloud to be detected. For example, the radar sensor may be a Millimeter-wave RADAR. Millimeter-wave radar can detect based on electromagnetic waves in the millimeter wave band. The frequency domain of electromagnetic waves in the millimeter wave band is generally 30 GHz (gigahertz) to 300 GHz; the wavelength is generally 1 mm to 10 mm. Alternatively, the radar sensor may also be a laser radar. This application is not limited to this.

[0067] It is understandable that there can be multiple visual sensors. For example, in the aforementioned digital intersection scene, visual sensors can be set at different locations around the road. By adjusting the shooting angles of these visual sensors, the collection areas of these visual sensors can all include the space area to be detected. In this way, images of the space area to be detected can be collected at different shooting angles by visual sensors set at different locations. Correspondingly, there are multiple images to be detected at this time.

[0068] By setting up visual sensors at different positions and collecting images of the spatial area to be detected at different shooting angles, the information of the spatial area to be detected contained in the images to be detected will be more comprehensive, which can reduce the probability of missed detection, false detection and other problems due to occlusion between objects, and improve the accuracy of subsequent target detection results.

[0069] For step S102, BEV refers to: a three-dimensional view drawn by looking down at the undulations of the ground from a high point. The electronic device can divide the spatial area to be detected into multiple BEV grids. One BEV grid corresponds to a cubic spatial area, and one BEV grid can be called a Pillar. The number of grids of the divided BEV grid can be determined by combining business needs and equipment performance. The more grids there are, the greater the amount of calculation when performing target detection, and the higher the accuracy of the target detection result; the fewer grids there are, the less the amount of calculation when performing target detection, and the shorter the time it takes to obtain the target detection result. Therefore, when the business needs require a shorter time to obtain the target detection result, a smaller number of grids can be set; when the equipment performance is good and the accuracy of the target detection result is required to be higher, a larger number of grids can be set. For example, the number of BEV grids obtained by division can be recorded as H×W. That is, on the BEV plane, the spatial area to be detected is divided into H rows of BEV grids, and each row of BEV grids includes W columns of BEV grids.

[0070] The point cloud to be detected includes multiple three-dimensional points. For each three-dimensional point, the electronic device can determine the Pillar to which the three-dimensional point belongs, that is, determine the BEV grid to which the three-dimensional point belongs. In this way, the electronic device divides the point cloud to be detected according to the BEV grid and determines the three-dimensional points contained in each BEV grid.

[0071] The coordinates of the three-dimensional point can be the coordinates in the world coordinate system. For example, the XOY plane of the world coordinate system can be the horizontal ground, the X-axis can be a regional boundary of the space area to be detected on the horizontal ground; the positive direction of the Z-axis can be the upward direction perpendicular to the horizontal ground. According to the right-hand rule of the three-dimensional coordinate system, the world coordinate system can be constructed. Obviously, the projections of the three-dimensional points belonging to the same BEV grid on the XOY plane belong to the same area. Therefore, for each three-dimensional point included in the point cloud to be detected, the electronic device can determine the area to which the projection of the three-dimensional point belongs only according to the coordinate value of the three-dimensional point on the X-axis and the coordinate value on the Y-axis, then the BEV grid corresponding to the area is the BEV grid to which the three-dimensional point belongs. It can reduce the amount of calculation when dividing the point cloud to be detected, reduce the time consumption of division, and thus reduce the time consumption of target detection and improve the efficiency of target detection.

[0072] In some embodiments, in the above-mentioned world coordinate system, if the radar sensor can collect the coordinate value of the three-dimensional point on the Z axis, the coordinates of each three-dimensional point included in the point cloud to be detected may include: the coordinate value on the X axis, the coordinate value on the Y axis, and the coordinate value on the Z axis.

[0073] In some embodiments, if the radar sensor fails to collect the coordinate value of the three-dimensional point on the Z axis, the coordinates of each three-dimensional point included in the point cloud to be detected may include: a coordinate value on the X axis and a coordinate value on the Y axis.

[0074] Alternatively, a preset virtual coordinate value may be used as the coordinate value of the coordinate of the three-dimensional point on the Z axis, such as being set to 0 or 1. In this case, the coordinate of each three-dimensional point includes: a coordinate value on the X axis, a coordinate value on the Y axis, and a preset virtual coordinate value.

[0075] For each BEV grid, the electronic device can extract features from the three-dimensional points contained in the BEV grid to obtain the initial point cloud features to be detected of the BEV grid. Exemplarily, the electronic device can process the coordinates of each three-dimensional point contained in the BEV grid based on PointNN (a deep learning algorithm for point cloud processing) to obtain the initial point cloud features to be detected of the BEV grid. The initial point cloud features to be detected of the BEV grid can also be called Pillar features.

[0076] Furthermore, the electronic device may also perform feature extraction on the image to be detected to obtain features of the initial image to be detected. Exemplarily, the electronic device may extract features of the initial image to be detected based on a convolutional neural network, such as ResNet (residual connection) or VGG (Visual Geometry Group Network), but is not limited thereto.

[0077] For step S103, when performing target detection, the electronic device can predetermine multiple reference three-dimensional points. Subsequently, by adjusting the coordinates of the reference three-dimensional points, the coordinates of the reference three-dimensional points are gradually approached to the position of the object in the spatial area to be detected. Therefore, in order to reduce the probability of missed detection, the number of reference three-dimensional points is usually much larger than the number of objects that may exist in the spatial area to be detected. And it can be understood that the more the number of reference three-dimensional points, the greater the amount of calculation when the electronic device performs target detection, and the longer the target detection takes. Therefore, according to the business needs of the application scenario, the number of reference three-dimensional points is usually set within a certain numerical range. For example, in the aforementioned digital intersection scenario, the number of reference three-dimensional points can be set to not less than 600 and not more than 1000.

[0078] For each reference 3D point, the electronic device can obtain the current coordinates and reference features of the reference 3D point. The reference feature can be a 256-dimensional vector. For example, the electronic device can randomly generate 256 numbers within a preset data range as the reference features of the reference 3D point. For example, the preset data range can be 0 to 100.

[0079] In one implementation, according to the size of the space area to be detected, the electronic device can directly generate coordinates belonging to the space area to be detected as the coordinates of the reference three-dimensional point. For example, in the aforementioned world coordinate system, a vertex of the space area to be detected is used as the coordinate origin; the length of the space area to be detected in the X-axis direction is 100 meters, the length in the Y-axis direction is 200 meters, and the length in the Z-axis direction is 50 meters. Then the electronic device can generate a random number in the interval [0, 100] as the coordinate of the reference three-dimensional point on the X-axis; generate a random number in the interval [0, 200] as the coordinate of the reference three-dimensional point on the Y-axis; generate a random number in the interval [0, 50] as the coordinate of the reference three-dimensional point on the Z-axis.

[0080] In one implementation, the electronic device can perform target detection on the spatial area to be detected based on the three-dimensional target detection model. The coordinates and reference features of the reference three-dimensional point can be adjusted during the training of the three-dimensional target detection model. The electronic device can directly obtain the coordinates and reference features of the reference three-dimensional point obtained by training the three-dimensional target detection model. For example, when training the three-dimensional target detection model, the electronic device can randomly determine three numbers in the interval [0, 1] as the current coordinate values ​​of the reference three-dimensional point on the X-axis, Y-axis, and Z-axis, respectively. The reference three-dimensional point determined at this time can be called the initial three-dimensional point. Then, the three-dimensional target detection model is trained based on the current coordinates of the reference three-dimensional point, and the current coordinates of the reference three-dimensional point are updated. After the training of the three-dimensional target detection model is completed, the current coordinates and reference features of the reference three-dimensional point, that is, the current coordinates and reference features of the reference three-dimensional point directly obtained by the electronic device. Subsequently, the electronic device can scale the current coordinates of the reference three-dimensional point according to the size of the spatial area to be detected. As in the above example, the coordinate value of the reference three-dimensional point on the X-axis is multiplied by 100; the coordinate value of the reference three-dimensional point on the Y-axis is multiplied by 200; the coordinate value of the reference three-dimensional point on the Z-axis is multiplied by 50. That is, the reference three-dimensional point is mapped to the spatial area to be detected. The coordinates of the initial three-dimensional point do not depend on the characteristics of the information collected by any sensor. Therefore, the three-dimensional target detection model trained based on the initial three-dimensional point can be adapted to different application scenarios and has high versatility. For the specific way in which the electronic device performs target detection in the spatial area to be detected based on the three-dimensional target detection model, please refer to the detailed description of the subsequent embodiments.

[0081] For each reference 3D point, according to the current coordinates of the reference 3D point, the electronic device can determine the BEV grid to which the reference 3D point currently belongs. According to the current reference features of the reference 3D point, the BEV grid is sampled from the BEV grid to which the reference 3D point currently belongs and other surrounding BEV grids, and then the initial point cloud features to be detected of the sampled BEV grids are fused to obtain the current fused point cloud features to be detected of the reference 3D point.

[0082] For step S104, the point cloud to be detected and the image to be detected are both acquired for the spatial area to be detected, so there is a mapping relationship between the three-dimensional points in the spatial area to be detected and the pixel coordinates in the image to be detected. The mapping relationship can be obtained by calibrating according to the acquisition method of the point cloud to be detected and the image to be detected. For example, the technician can place a calibration plate in the spatial area to be detected. The calibration point cloud of the calibration plate is collected by the acquisition method of the point cloud to be detected, and the calibration image of the calibration plate is collected by the acquisition method of the image to be detected. For each mark point on the calibration plate, according to the coordinates of the three-dimensional point corresponding to the mark point in the calibration point cloud, and the pixel coordinates corresponding to the mark point in the calibration image, the mapping relationship between the three-dimensional point in the spatial area to be detected and the pixel coordinates of the image to be detected can be solved.

[0083] When there are multiple images to be detected, for each image to be detected, calibration is performed according to the point cloud to be detected and the acquisition method of the image to be detected, and the mapping relationship between the pixel coordinates in the image to be detected and the three-dimensional points in the spatial area to be detected can be obtained. In addition, the initial image features to be detected are obtained by feature extraction of the image to be detected, so there is also a mapping relationship between the feature values ​​in the initial image features to be detected and the pixel coordinates in the image to be detected.

[0084] That is, a 3D point in the space area to be detected corresponds to a pixel coordinate. The portion of the pixel coordinate corresponding to the initial image feature to be detected is the portion of the 3D point corresponding to the initial image feature to be detected.

[0085] Exemplarily, a feature value in the features of the initial image to be detected is obtained by extracting features from pixels in an image area in the image to be detected. Then, the corresponding part of a three-dimensional point in the initial image to be detected is the feature value obtained by extracting features from the pixels corresponding to the three-dimensional point in the image to be detected.

[0086] In this way, for each reference three-dimensional point, the electronic device can obtain the portion of the reference three-dimensional point corresponding to the initial image feature to be detected, and obtain the current intermediate image feature to be detected of the reference three-dimensional point.

[0087] For step S105 and step S106, for each reference 3D point, the electronic device can obtain the current fused point cloud feature to be detected and the current intermediate image feature to be detected of the reference 3D point. Then, the feature values ​​of the current fused point cloud feature to be detected of each reference 3D point and the feature values ​​of the current intermediate image feature to be detected of each reference 3D point are spliced ​​to obtain the current spliced ​​feature to be detected.

[0088] Specifically, if the current fused point cloud feature to be detected of the reference three-dimensional point i is recorded as Ri, the current intermediate image feature to be detected is recorded as Ci, and the total number of reference three-dimensional points is recorded as I, then the electronic device can concat the feature values ​​of the current fused point cloud feature to be detected of the reference three-dimensional point and the feature values ​​of the current intermediate image features to be detected of each reference three-dimensional point, and the obtained features can be recorded as: {R1, R2, R3, …, RI, C1, C2, C3, …, CI}.

[0089] Then, the features obtained by concat are encoded to obtain the current splicing features to be detected. Exemplarily, the electronic device can use a convolutional neural network to jointly encode the features obtained by concat to obtain the current splicing features to be detected.

[0090] By decoding the current splicing features to be detected, the current coordinate adjustment amount of each reference 3D point and the current reference feature adjustment amount of each reference 3D point can be obtained. For each reference 3D point, the sum of the current coordinates and the coordinate adjustment amount of the reference 3D point is calculated to obtain the current coordinates of the reference 3D point; the sum of the current reference feature of the reference 3D point and the reference feature adjustment amount is calculated to obtain the current reference feature of the reference 3D point. For example, the electronic device can decode the current splicing features to be detected based on the Decoder in the deep learning model. For example, the decoder can be a decoding module in the Transformer for decoding processing. In this way, the electronic device also performs a decoding and updates the coordinates and reference features of each reference 3D point.

[0091] Then, the electronic device can use the current coordinates and reference features of each reference 3D point updated by this decoding to return to the step of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point. That is, based on the current coordinates and reference features of each reference 3D point, the current splicing features to be detected are determined again, so that the next decoding is performed based on the current splicing features to be detected, and the coordinates and reference features of each reference 3D point are updated again until the number of decodings reaches the preset number. The preset number can be determined based on the type of decoder used. For example, when the decoding module in the Transformer is used as a decoder, since the Transformer usually includes 6 decoding modules, the preset number can be 6.

[0092] For step S107, after the number of decodings reaches the preset number, for each reference 3D point, the current coordinates of the reference 3D point are the position of the object in the detected spatial area (hereinafter referred to as the position to be screened). It can be understood that different reference 3D points can indicate the same object, that is, one object may correspond to multiple positions to be screened. Therefore, the electronic device can post-process the position to be screened to obtain the position of the object in the spatial area to be detected.

[0093] In some embodiments, the decoding result may also include the confidence of the reference three-dimensional point, and the confidence may indicate the probability that an object exists at the position indicated by the reference three-dimensional point. When the confidence of a reference three-dimensional point is greater than the confidence threshold, it indicates that the probability that an object exists at the reference three-dimensional point is high, and the coordinates of the reference three-dimensional point may be used as the position where the object exists in the detected spatial area to be detected, as the target detection result of the spatial area to be detected. The confidence threshold may be determined according to the detection accuracy requirements in the actual scene. The higher the confidence threshold, the higher the accuracy of the detected object; the lower the confidence threshold, the lower the probability of problems such as missed detection. When the accuracy of the detected object is required to be high, the confidence threshold may be set to a larger value, such as 0.95; when the probability of problems such as missed detection is required to be reduced, the confidence threshold may be set to a smaller value, such as 0.7.

[0094] In some embodiments, the detected position of the object may indicate: the coordinates of the center point of the minimum circumscribed cube of the object, the size of the minimum circumscribed cube of the object, and the orientation of the object. If the detected position of the object may include the above information, the electronic device may subsequently calculate the vertex coordinates of the minimum circumscribed cube of the object based on the above information, that is, determine the position of the object. Alternatively, the detected position of the object may be the vertex coordinates of the minimum circumscribed cube of the object, that is, the electronic device may directly obtain the position of the object.

[0095] In some embodiments, the decoding result may further include: the category of the object existing at the reference 3D point. Then, according to the decoding result, the electronic device may also obtain the category of the detected object to meet the business needs in the actual scene.

[0096] In some embodiments, the electronic device may perform target detection on the spatial area to be detected based on an end-to-end three-dimensional target detection model. Figure 1 Based on Figure 2 , the above step S102 may include the following steps:

[0097] S1021: Divide the point cloud to be detected according to the bird's-eye view grids, and determine the three-dimensional points to be detected contained in each bird's-eye view grid obtained by the division; use the point cloud feature extraction network in the pre-trained three-dimensional target detection model to extract features from the information to be used of the three-dimensional points to be detected contained in each bird's-eye view grid to obtain the initial point cloud features to be detected of each bird's-eye view grid; and input the image to be detected into the image feature extraction network in the three-dimensional target detection model to obtain multiple initial image features to be detected.

[0098] Among them, the information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point, and the coordinates of the center of mass of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model is obtained by training based on sample images, sample point clouds and sample labels representing the positions of objects in the sample space area; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder.

[0099] Accordingly, the above-mentioned step S103 may include the following steps: S1031: for each reference three-dimensional point, input the current reference feature of the reference three-dimensional point into the deformable attention network to obtain a preset sampling number of position offsets and a weight corresponding to each position offset; for each position offset, use the position offset to offset the current coordinates of the reference three-dimensional point, and determine the bird's-eye view grid to which the offset coordinates belong, as the bird's-eye view grid corresponding to the position offset; according to the weight corresponding to each position offset, fuse the initial point cloud features to be detected of the bird's-eye view grids corresponding to each position offset to obtain the current fused point cloud features to be detected of the reference three-dimensional point.

[0100] Accordingly, the above step S104 may include the following steps:

[0101] S1041: Input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each initial image feature to be detected; according to the weight of each initial image feature to be detected, fuse the corresponding feature values ​​of the reference 3D point in each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference 3D point.

[0102] Accordingly, the above step S106 may include the following steps:

[0103] S1061: Input the current splicing features to be detected into the decoder to obtain the current decoding result, and update the current coordinates and reference features of the reference three-dimensional point based on the current decoding result; return to execute the steps of sampling the bird's-eye view grid within the neighborhood of each reference three-dimensional point based on the current coordinates and reference features of the reference three-dimensional point, and fusing the sampled initial point cloud features to be detected of the bird's-eye view grid to obtain the current fused point cloud features to be detected of the reference three-dimensional point, until the number of decoding times reaches the preset number.

[0104] Since the electronic device will subsequently use the point cloud feature extraction network in the pre-trained three-dimensional target detection model to extract features from the information to be used of the three-dimensional points to be detected contained in each BEV grid. It can be understood that the format of the input data of the point cloud feature extraction network is set. For example, the input data of the point cloud feature extraction network is a tensor of (D, N, P), where D represents the number of information dimensions of the information to be used of a three-dimensional point. For example, when the information to be used of a three-dimensional point includes the coordinates of the three-dimensional point and the coordinates of the center of mass of the BEV grid to which the three-dimensional point belongs, D can be 6. N represents the number of three-dimensional points to be detected in each BEV grid (i.e., the specified number in subsequent embodiments). P represents the number of BEV grids obtained by division, i.e., the product of the aforementioned H and W.

[0105] For each BEV grid, after determining the three-dimensional points included in the BEV grid, the electronic device can determine the minimum circumscribed cube of the three-dimensional points included in the BEV grid, and then use the centroid of the minimum circumscribed cube as the centroid of the BEV grid.

[0106] Since the number of three-dimensional points contained in each BEV grid may be different, in order to obtain the initial point cloud features to be detected of each BEV grid through the point cloud feature extraction network, the electronic device can determine the three-dimensional points to be detected contained in the BEV grid according to the number of three-dimensional points contained in the BEV grid.

[0107] In some embodiments, the aforementioned determination of the three-dimensional points to be detected contained in each BEV grid obtained by division may include the following steps: for each BEV grid obtained by division, if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than a specified number, then a specified number of three-dimensional points are sampled from the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is equal to the specified number, then the three-dimensional points contained in the BEV grid are used as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, then the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is calculated to generate a three-dimensional point whose coordinates are preset values ​​of the difference; combining the generated three-dimensional points and the three-dimensional points in the point cloud to be detected contained in the BEV grid, the three-dimensional points to be detected contained in the BEV grid are obtained.

[0108] For each BEV grid obtained by division, when the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than the specified number, that is, the information to be used of the three-dimensional points contained in the BEV grid does not need to be fully input into the point cloud feature extraction network. At this time, the electronic device can sample the three-dimensional points contained in the BEV grid, and use the sampled specified number of three-dimensional points as the three-dimensional points to be detected contained in the BEV grid. Correspondingly, in the input data of the point cloud feature extraction network, the data corresponding to the BEV grid is the sampled information to be used of the three-dimensional points to be detected.

[0109] When the number of 3D points in the point cloud to be detected contained in the BEV grid is equal to the specified number, the information to be used of all 3D points contained in the BEV grid is input into the point cloud feature extraction network. At this time, the electronic device can directly determine all 3D points contained in the BEV grid as 3D points to be detected. Correspondingly, in the input data of the point cloud feature extraction network, the data corresponding to the BEV grid is the information to be used of all 3D points in the BEV grid.

[0110] In the case where the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, even if all the three-dimensional points contained in the BEV grid are determined as three-dimensional points to be detected, the data corresponding to the BEV grid in the above-mentioned set data format cannot be obtained. At this time, the electronic device can perform supplementary processing on the data corresponding to the BEV grid. Specifically, the electronic device can calculate the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid (hereinafter referred to as the number of groups to be supplemented), and the number of groups to be supplemented is the number of data groups that need to be supplemented with the data corresponding to the BEV grid. Then generate three-dimensional points with preset coordinates of the number of groups to be supplemented. When all values ​​in the information to be used of a three-dimensional point are preset values, the three-dimensional point can be indicated to be used to fill the input data so that the input data has a set format. For example, the preset value can be 0 or 1.

[0111] For example, if the data to be used of a 3D point is represented as: (x, y, z, xc, yc, zc), where (x, y, z) represents the coordinates of the 3D point; (xc, yc, zc) represents the coordinates of the centroid of the BEV grid to which the 3D point belongs. When the preset value is 0, the information to be used of the 3D point generated by the electronic device based on the preset value can be (0, 0, 0, 0, 0, 0).

[0112] Furthermore, for the BEV grid, the electronic device can use the generated three-dimensional points and the three-dimensional points in the point cloud to be detected contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid. Correspondingly, in the input data of the point cloud feature extraction network, the data corresponding to the BEV grid is: the information to be used of all three-dimensional points in the BEV grid and the information to be used of three-dimensional points generated based on preset values.

[0113] Based on the above processing, when the number of three-dimensional points contained in a BEV grid is not the specified number, the electronic device can determine the three-dimensional points to be detected contained in the BEV grid based on the number of three-dimensional points contained in the BEV grid, based on sampling or supplementary processing. Subsequently, the point cloud feature extraction network can be used to extract features from the information to be used of the three-dimensional points to be detected contained in each BEV grid, thereby reducing the probability of problems such as the low accuracy of the extracted initial point cloud features to be detected of each BEV grid due to the inconsistent number of three-dimensional points contained in each BEV grid, and improving the accuracy of the target detection results obtained based on the initial point cloud features to be detected of each BEV grid.

[0114] In some embodiments, the information to be utilized of a three-dimensional point also includes at least one of the following: the coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the real position corresponding to the detected three-dimensional point.

[0115] When the information to be used of a 3D point also includes: the coordinate offset of the 3D point relative to the centroid of the BEV grid to which the 3D point belongs, that is, the input data of the point cloud feature extraction network carries data indicating the relationship between some input data, the difficulty of the point cloud feature extraction network to learn the relationship between the input data will be lower, and the learned relationship between the input data will be more accurate. In this case, the initial point cloud features to be detected of each BEV grid will be more accurate.

[0116] When the information to be used of a three-dimensional point also includes: the feature data at the real position corresponding to the detected three-dimensional point, the point cloud feature extraction network performs feature extraction based on the feature data at each real position actually detected, and the extracted point cloud features to be detected of each BEV grid are more in line with the actual scene, that is, more accurate. Then the accuracy of the target detection results obtained based on the more accurate point cloud features to be detected of each BEV grid will be higher. For example, the feature data may be the signal-to-noise ratio, reflection intensity, etc. In the case where the radar sensor can detect a variety of feature data at the real position corresponding to the three-dimensional point, the information to be used of the three-dimensional point may also include the multiple feature data to further improve the accuracy of the extracted initial point cloud features to be detected of each BEV grid.

[0117] Exemplarily, when the information to be used of a 3D point also includes: the coordinate offset of the 3D point relative to the centroid of the BEV grid to which the 3D point belongs, and the feature data detected at the real position corresponding to the 3D point, the data to be used of a 3D point can be expressed as: (x, y, z, r, xc, yc, zc, xp, yp), at which time, the information dimension number of the information to be used of the 3D point is 9, that is, when the input data of the point cloud feature extraction network is a tensor of (D, N, P), D is 9. Among them, (x, y, z) represents the coordinates of the 3D point; r represents the feature data at the real position corresponding to the 3D point; (xc, yc, zc) represents the coordinates of the centroid of the BEV grid to which the 3D point belongs; (xp, yp) represents the coordinate offset of the 3D point relative to the centroid of the BEV grid to which the 3D point belongs.

[0118] In some embodiments, when the radar sensor can collect the coordinate value of the three-dimensional point on the Z axis, the coordinate value of the three-dimensional point on the Z axis contained in the to-be-used information of a three-dimensional point may be: the real coordinate value collected by the radar sensor. When the radar sensor fails to collect the coordinate value of the three-dimensional point on the Z axis, a preset virtual coordinate value may be used as the coordinate value of each three-dimensional point on the Z axis. For example, the preset virtual coordinate value may be set to 0 or 1.

[0119] After acquiring the image to be detected, the electronic device can use the image feature extraction network to extract features of the image to be detected to obtain multiple initial image features to be detected. For example, the image feature extraction network may include ResNet, VGG, etc. in the aforementioned embodiments, but is not limited thereto.

[0120] In some embodiments, the image feature extraction network includes multiple downsampling layers connected in series; there is one image to be detected, and the multiple initial image features to be detected include: image features of the image to be detected extracted by multiple downsampling layers; or, the image to be detected includes images of the spatial area to be detected captured at multiple different shooting angles, and the multiple initial image features to be detected include image features of each image to be detected.

[0121] After the image to be detected is input into the first downsampling layer in the serial connection, the multiple downsampling layers can sequentially downsample their own input data, and the output data of different downsampling layers, that is, the image features of the image to be detected extracted by different downsampling layers, are different initial image features to be detected. Obviously, different initial image features to be detected include different numbers of feature values. Taking the downsampling layer as a convolution layer, the convolution kernel size is 3×3, and the step size is 1 as an example, for image 1 containing 5×5 pixels, the image 1 is convolved once by the first downsampling layer to obtain an image feature a of size 3×3, that is, the output data of the first downsampling layer; the image feature a is convolved once by the second downsampling layer to obtain an image feature b of size 1, that is, the output data of the second downsampling layer. The above-mentioned image feature a and image feature b are the image features of image 1 extracted by multiple downsampling layers.

[0122] Therefore, when there is only one image to be detected, multiple initial image features to be detected can be extracted through the multiple downsampling layers connected in series included in the image feature extraction network.

[0123] When there are multiple images to be detected, that is, when multiple visual sensors collect images of the spatial area to be detected from different shooting angles, for each image to be detected, an image feature of the image to be detected can be extracted through an image feature extraction network; or, in accordance with the aforementioned method of extracting multiple initial image features to be detected of an image to be detected, multiple initial image features to be detected of each image to be detected can be extracted respectively.

[0124] Based on the above processing, multiple initial image features to be detected of the image to be detected are extracted through the image feature extraction network, that is, richer initial image features to be detected that contain more information can be obtained. Then, the target detection result is obtained based on the initial image features to be detected that contain more information, that is, the target detection result is obtained by integrating more information, which can improve the accuracy of the obtained target detection result.

[0125] For each reference 3D point, the electronic device can input the current reference feature of the reference 3D point into DAT (Deformable Attention Transformer) and obtain the preset number of position offsets to be obtained (i.e., the preset number of samples). The deformable attention network can process the current reference feature of the reference 3D point based on the deformable attention mechanism to obtain a preset number of sampled position offsets and a weight corresponding to each position offset. For each position offset, the electronic device can use the position offset to offset the current coordinates of the reference 3D point.

[0126] In the case where the electronic device directly generates the coordinates belonging to the spatial area to be detected as the coordinates of the initial three-dimensional point, the electronic device can directly calculate the sum of the current coordinates of the reference three-dimensional point and the position offset, and the calculated coordinates are the offset coordinates. In the case where the coordinates of the reference three-dimensional point are obtained by the electronic device based on a number randomly determined in the interval [0, 1], the electronic device can first scale the coordinates of the reference three-dimensional point according to the size of the spatial area to be detected, and then calculate the sum of the scaled coordinates and the position offset, and the calculated coordinates are the offset coordinates.

[0127] The BEV grid to which the offset coordinate belongs is the BEV grid corresponding to the position offset, that is, the BEV grid sampled based on the position offset. According to the weights corresponding to each position offset, the weighted sum of the initial point cloud features to be detected of the BEV grid corresponding to each position offset is calculated to obtain the current fused point cloud features to be detected of the reference 3D point.

[0128] Exemplarily, the electronic device may calculate the current to-be-detected fused point cloud feature of the reference 3D point based on the following formula (1):

[0129] ; (1)

[0130] in, Represents the current fused point cloud feature to be detected of the i-th reference 3D point; represents the initial point cloud features to be detected of the BEV grid corresponding to the current k-th position offset of the i-th reference 3D point; Represents the weight corresponding to the current k-th position offset of the i-th reference 3D point; Indicates the current coordinates of the i-th reference 3D point; Indicates the coordinates obtained by scaling the current coordinates of the i-th reference 3D point according to the size of the spatial area to be detected; represents the current k-th position offset of the i-th reference 3D point; K represents the preset number of samples.

[0131] For each reference 3D point, the electronic device can input the current reference feature of the reference 3D point into the weighted attention network. The weighted attention network processes the current reference feature of the reference 3D point based on the Attention mechanism to obtain the weight of each initial image feature to be detected. After the electronic device determines the corresponding part of the reference 3D point in each initial image feature to be detected, that is, determines the corresponding feature value of the reference 3D point in each initial image feature to be detected, it can calculate the weighted sum of the determined feature values ​​according to the weights of each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference 3D point.

[0132] In some embodiments, the three-dimensional object detection model also includes an MLP (Multilayer Perceptron).

[0133] In the aforementioned step, for each reference 3D point, the current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and the weight corresponding to each position offset. The method may also include the following steps: Step 1: Input the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into the multi-layer perceptron to obtain the correction amount for correcting the initial conversion matrix. The initial conversion matrix is ​​obtained by calibration according to the acquisition method of the point cloud to be detected and the image to be detected.

[0134] Step 2: Use the obtained correction amount to correct the initial transformation matrix.

[0135] Correspondingly, the aforementioned steps fuse the eigenvalues ​​corresponding to the reference 3D point in each of the initial image features to be detected according to the weights of each of the initial image features to be detected, and obtain the current intermediate image features to be detected of the reference 3D point, which may include the following steps: Step 3: Use the corrected transformation matrix to transform the current coordinates of the reference 3D point, and obtain the pixel coordinates corresponding to the reference 3D point in the image to be detected as the reference coordinates to be detected. Step 4: According to the weights of each of the initial image features to be detected, fuse the eigenvalues ​​corresponding to the reference coordinates to be detected in each of the initial image features to be detected, and obtain the current intermediate image features to be detected of the reference 3D point.

[0136] The above calibration is performed according to the acquisition method of the point cloud to be detected and the image to be detected: the mapping relationship between the three-dimensional points in the space area to be detected and the pixel coordinates in the image to be detected, that is, the initial transformation matrix. The initial transformation matrix can be recorded as That is, the initial transformation matrix is ​​a 3×3 matrix.

[0137] It can be understood that the initial point cloud features to be detected of a BEV grid include multiple eigenvalues, and the initial image features to be detected also include multiple eigenvalues. The electronic device can concat the initial point cloud features to be detected of each BEV grid and the initial image features to be detected. Then, the concat features are input into the MLP to obtain a 3×3 matrix output by the MLP as the correction amount for correcting the initial transformation matrix.

[0138] In some embodiments, the aforementioned inputting of the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into a multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix may include the following steps: inputting the initial point cloud features to be detected of each BEV grid, the initial image features to be detected, and the current coordinates of each reference three-dimensional point into a multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0139] It is understandable that at this time, decoding based on the reference 3D points has not been performed, and the coordinates of the reference 3D points have not been updated. In the case where the electronic device generates the coordinates of the reference 3D points, the current coordinates of each reference 3D point are the coordinates generated by the electronic device; in the case where the electronic device performs target detection on the detection space area based on the 3D target detection model, the current coordinates of each reference 3D point are the current coordinates of each reference 3D point after the 3D target detection model training is completed.

[0140] The electronic device can concat the initial point cloud features to be detected of each BEV grid, multiple initial image features to be detected, and the current coordinates of each reference three-dimensional point. The concat features are then input into the MLP to obtain a 3×3 matrix output by the MLP as the correction amount for correcting the initial transformation matrix. In this way, the current coordinates of the reference three-dimensional points that indicate the position of the object in the spatial area to be detected are comprehensively determined to determine the correction amount, and the correction amount can indicate the position of the object in the spatial area to be detected. After the initial transformation matrix is ​​corrected based on the correction amount, the position of the existing object indicated by the reference position to be detected obtained based on the corrected transformation matrix is ​​more accurate, thereby improving the accuracy of the subsequent target detection results.

[0141] Then, the electronic device can use the obtained correction amount to correct the initial conversion matrix. That is, for each element in the initial conversion matrix, calculate the sum of the element and the element at the same position of the element in the obtained correction amount, and update the element. The elements in the initial conversion matrix are updated, that is, the initial conversion matrix is ​​corrected using the obtained correction amount. Furthermore, for each reference three-dimensional point, the electronic device can use the corrected conversion matrix to transform the coordinates of the reference three-dimensional point to obtain the pixel coordinates in the pixel coordinate system, that is, obtain the corresponding reference coordinates to be detected in the image to be detected. The eigenvalues ​​corresponding to the reference coordinates to be detected in each feature of the initial image to be detected, that is, the eigenvalues ​​corresponding to the reference three-dimensional point in each feature of the initial image to be detected.

[0142] As mentioned above, there is a mapping relationship between the feature value in the initial image feature to be detected and the pixel point in the image to be detected. After obtaining the corresponding reference coordinate to be detected in the image to be detected, the electronic device can determine the feature value corresponding to the reference coordinate to be detected in the initial image feature to be detected according to the above mapping relationship.

[0143] For example, if the number of pixels contained in the image to be detected is 1000×1000, the initial image features to be detected include: the initial image feature 1 to be detected includes 200×200 eigenvalues; the initial image feature 2 to be detected includes 100×100 eigenvalues; the initial image feature 3 to be detected includes 20×20 eigenvalues, etc. Correspondingly, at this time, if the reference coordinates to be detected are (100, 100), the eigenvalues ​​corresponding to the reference coordinates to be detected in the initial image features to be detected include: the eigenvalue at the coordinate (20, 20) in the initial image feature 1 to be detected; the eigenvalue at the coordinate (10, 10) in the initial image feature 2 to be detected; the eigenvalue at the coordinate (2, 2) in the initial image feature 3 to be detected.

[0144] Furthermore, according to the weights of the initial image features to be detected, the feature values ​​corresponding to the reference coordinates to be detected in the initial image features to be detected are fused to obtain the current intermediate image features to be detected of the reference three-dimensional point.

[0145] Based on the above processing, the electronic device can correct the initial conversion matrix obtained by calibration based on the initial point cloud features to be detected of each BEV grid and the initial image features to be detected. That is, the initial conversion matrix is ​​corrected based on the actual application scenario, and the corrected conversion matrix is ​​more in line with the actual application scenario than the initial conversion matrix. The current intermediate image features to be detected based on the corrected conversion matrix are more accurate than the current intermediate image features to be detected based on the initial conversion matrix, so that subsequent target detection based on the more accurate current intermediate image features to be detected can improve the accuracy of the target detection results.

[0146] Exemplarily, the electronic device may obtain a corrected correction matrix based on the following formula (2):

[0147] ; (2)

[0148] in, represents the correction matrix after correction; , represents the initial transformation matrix; represents the correction amount obtained by the multilayer perceptron; , represents the coordinates of each reference 3D point, and M represents the number of reference 3D points; , represents the initial image feature to be detected, C1 represents the number of feature channels of the initial image feature to be detected; , represents the initial point cloud features to be detected of each BEV grid, and C2 represents the number of feature dimensions of the initial point cloud features to be detected of each BEV grid.

[0149] Accordingly, the electronic device can calculate the current to-be-detected intermediate image feature of the reference 3D point based on the following formula (3):

[0150] ; (3)

[0151] in, Represents the current intermediate image feature to be detected of the i-th reference 3D point; Indicates the current coordinates of the i-th reference 3D point; Indicates the reference coordinates to be detected corresponding to the current coordinates of the i-th reference 3D point in the n-th image to be detected; Represents: the reference coordinates to be detected extracted by the jth downsampling layer Corresponding image features; represents the weight of the image feature extracted by the jth downsampling layer in the nth image to be detected of the i-th reference 3D point, m represents the number of serially connected downsampling layers included in the image feature extraction network; N represents the number of images to be detected.

[0152] In some embodiments, based on the weighted attention network, the electronic device can obtain the weight of each image to be detected. Furthermore, for each image to be detected, the electronic device can calculate the quotient of the weight of the image to be detected and the number of downsampling layers, and use the calculation result as the weight of each of the multiple image features of the image to be detected extracted through multiple downsampling layers.

[0153] In some embodiments, since the initial image features to be detected and the initial point cloud features to be detected of each BEV grid need to be input into the MLP to obtain the correction amount for correcting the initial transformation matrix, that is, the initial point cloud features to be detected of each BEV grid also need to have a specified format, and the format is related to the format of the initial point cloud to be detected. For example, the electronic device can perform dimension upgrade on the features output by the point cloud feature extraction network to obtain a tensor with a dimension of (C, N, P); the dimension upgraded tensor is pooled in the dimension of N to obtain a tensor with a dimension of (C, P). And as mentioned above, P is the product of H and W, then the tensor with a dimension of (C, P) can be transformed into a tensor with a dimension of (C, H, W) as the initial point cloud features to be detected of each BEV grid. Wherein, C is the number of output channels of the image feature extraction network.

[0154] For each reference 3D point, the reference coordinates to be detected obtained by performing coordinate transformation on the current coordinates of the reference 3D point may not be integers, that is, the reference coordinates to be detected may not correspond to the pixels in the image to be detected.

[0155] In some embodiments, when the reference coordinates to be detected are not integers, the electronic device can round the non-integer coordinate values ​​of the reference coordinates to be detected, and use the feature values ​​corresponding to the coordinates obtained after rounding in each initial image feature to be detected as the feature values ​​corresponding to the reference coordinates to be detected.

[0156] In some embodiments, before the eigenvalues ​​corresponding to the reference coordinates to be detected in each initial image feature to be detected are fused according to the weights of each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference three-dimensional point, the method may further include the following steps: if the reference coordinate to be detected is not an integer, then other pixel coordinates within the neighborhood of the reference coordinate to be detected are obtained; for each initial image feature to be detected, the eigenvalues ​​corresponding to the other obtained pixel coordinates in the initial image feature to be detected are interpolated as the eigenvalues ​​corresponding to the reference coordinate to be detected in the initial image feature to be detected.

[0157] The electronic device can obtain other pixel coordinates within the neighborhood of the reference coordinate to be detected, such as rounding up and rounding down the coordinate values ​​that are not integers in the reference coordinate to be detected, and determining the pixel coordinates at the coordinate values ​​determined by rounding as other pixel coordinates within the neighborhood of the reference coordinate to be detected. For example, when the reference coordinate to be detected is (1.5, 3.5), 1.5 is rounded up to obtain 2; 1.5 is rounded down to obtain 1; 3.5 is rounded up to obtain 4; 3.5 is rounded down to obtain 3. Then the other pixel coordinates within the neighborhood of the reference coordinate to be detected include: (1, 3), (2, 3), (1, 4), (2, 4).

[0158] At this time, the other pixel coordinates correspond to characteristic values ​​in each initial image feature to be detected. For each initial image feature to be detected, the electronic device can interpolate the characteristic values ​​corresponding to the other pixel coordinates obtained in the initial image feature to be detected as the characteristic values ​​corresponding to the reference coordinates to be detected in the initial image feature to be detected. For example, the electronic device can interpolate the obtained characteristic values ​​using a bilinear interpolation method.

[0159] Based on the above processing, when the reference coordinate to be detected is not an integer, the electronic device comprehensively considers the coordinates of other pixels within the neighborhood of the reference coordinate to be detected, and jointly determines the eigenvalue corresponding to the reference coordinate to be detected. The eigenvalue corresponding to the reference coordinate to be detected determined in this way will be more accurate, and the current intermediate image feature to be detected of the reference three-dimensional point subsequently calculated based on the more accurate eigenvalue will be more accurate, which can improve the accuracy of the subsequent target detection results obtained based on the current intermediate image feature to be detected.

[0160] After the electronic device splices the current fused point cloud features to be detected and the intermediate image features to be detected of each reference three-dimensional point to obtain the current spliced ​​features to be detected, the current spliced ​​features to be detected can be input into the decoder to obtain the current decoding result. The decoder can be a decoder based on a deep learning model in the aforementioned embodiment, such as a decoding module in a Transformer. The decoding result may include the coordinate offset of each reference three-dimensional point, the reference feature offset of each reference three-dimensional point, etc. Based on the offset obtained by decoding, the electronic device can update the current coordinates and reference features of each reference three-dimensional point.

[0161] In some embodiments, the decoding result may also include a size offset of the size of the minimum circumscribed cube of the object at the reference 3D point, a category offset of the category of the object at the reference 3D point, etc. The electronic device may use the decoding result to update the size of the minimum circumscribed cube of the object at the reference 3D point, the category of the object at the reference 3D point, etc.

[0162] Based on the above processing, electronic devices can perform target detection based on an end-to-end trained three-dimensional target detection model, thereby reducing maintenance costs and improving the versatility of target detection methods.

[0163] In some embodiments, see Figure 3 , Figure 3 A schematic diagram of a target detection method provided in an embodiment of the present application. In an embodiment of the present application, the three-dimensional target detection model is divided into four modules according to its functions. Among them, the multi-sensor feature extraction module 301 includes an image backbone network (i.e., the image feature extraction network in the aforementioned embodiment), and a radar backbone network (i.e., the point cloud feature extraction network in the aforementioned embodiment). The learnable radar matrix mapping sampling module 302 includes the MLP in the aforementioned embodiment. The query vector-based learnable bird's-eye view feature extraction module 303 includes the DAT and weighted attention network in the aforementioned embodiment. The three-dimensional detection decoder module 304 includes the decoder in the aforementioned embodiment. The reference three-dimensional point can also be called an anchor point; the reference feature of each reference three-dimensional point can be recorded as . Where M represents the number of reference 3D points, Represents the current reference feature of the i-th reference 3D point, and C represents the number of output channels of the image feature extraction network. The coordinates and reference features of a reference 3D point can be called a 3D Query (three-dimensional query vector). According to the correlation between the Query (query vector) and the Key (key), the corresponding Value (value) is found, and the weight of each Value can be obtained. Subsequently, based on the weight of each Value, the fusion result can be calculated. In the process of obtaining the weight of each position offset based on DAT in the present application, each position offset is Value, and the weight of each position offset is the weight of the Value obtained based on the correlation between Query and Key; in the process of obtaining the weight of each initial image feature to be detected based on the Attention mechanism, each initial image feature to be detected is Value, and the weight of each initial image feature to be detected is the weight of the Value obtained based on the correlation between Query and Key.

[0164] After acquiring the single-camera / multi-camera image (i.e., one or more images to be detected in the aforementioned embodiment, hereinafter referred to as camera image) and the radar point cloud (i.e., the point cloud to be detected in the aforementioned embodiment), the electronic device can input the camera image and the radar point cloud into the multi-sensor feature extraction module 301. The image features of the camera image (i.e., the initial image features to be detected in the aforementioned embodiment) are extracted through the image backbone network included in the multi-sensor feature extraction module 301; and the grid features of the radar point cloud (i.e., the initial point cloud features to be detected of each BEV grid in the aforementioned embodiment) are extracted through the radar backbone network.

[0165] The aforementioned extraction of the point cloud features to be detected of each BEV grid based on the point cloud feature extraction network can also be referred to as the extraction of the Pillar features of each grid using the method of constructing a Pillar grid. Specifically, the electronic device can divide the radar point cloud to determine the three-dimensional points contained in each of the H×W BEV grids; for each BEV grid, the three-dimensional points to be detected in the BEV grid are determined according to the number of three-dimensional points contained in the BEV grid; and the radar backbone network is used to extract features of the information to be used of the three-dimensional points to be detected in the BEV grid. The extracted features are dimensionally upgraded to obtain a tensor with a dimension of (C, N, P); pooling is performed in the dimension of N to obtain a tensor with a dimension of (C, P); the tensor with a dimension of (C, P) is transformed into a tensor with a dimension of (C, H, W) as the grid feature of each BEV grid.

[0166] The electronic device uses a learnable radar-view matrix mapping sampling module 302 to concat the extracted image features, grid features, and each three-dimensional coordinate (i.e., the coordinates of each reference three-dimensional point in the aforementioned embodiment), and then uses MLP to extract features from the concat features to obtain a correction amount for correcting the initial transformation matrix. Then, the obtained correction amount is used to correct the RV (Radar-View) matrix (i.e., the initial transformation matrix in the aforementioned embodiment). The transformation matrix is ​​a homography matrix that can mark the mapping relationship between the BEV space and the image pixel position.

[0167] For example, see Figure 4 , Figure 4 A schematic diagram of a learnable radar vision matrix mapping sampling module 302 provided in an embodiment of the present application. The radar features (i.e., the initial point cloud features to be detected of each BEV grid in the aforementioned embodiment), three-dimensional coordinates, and image features are input into the learnable multi-layer perceptron module, and the radar vision matrix is ​​corrected using the output data of the learnable multi-layer perceptron module to obtain an optimized radar vision matrix (i.e., the corrected conversion matrix in the aforementioned embodiment).

[0168] Furthermore, through the query vector-based learnable bird's-eye view feature extraction module 303, for each reference three-dimensional point, the current reference feature of the reference three-dimensional point is input into the DAT included in the query vector-based learnable bird's-eye view feature extraction module 303 to obtain a preset sampling number of position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference three-dimensional point are offset using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; according to the weight corresponding to each position offset, the initial point cloud features to be detected of the BEV grids corresponding to each position offset are feature fused to obtain the current fused point cloud features to be detected of the reference three-dimensional point.

[0169] Using the weighted attention network included in the query vector-based learnable bird's-eye view feature extraction module 303, for each reference three-dimensional point, the current reference feature of the reference three-dimensional point is input into the weighted attention network to obtain the weight of each initial image feature to be detected; according to the weight of each initial image feature to be detected, the feature values ​​corresponding to the reference coordinates to be detected in each initial image feature to be detected are feature fused to obtain the current intermediate image feature to be detected of the reference three-dimensional point.

[0170] Then, the electronic device splices the current fused point cloud features to be detected and the intermediate image features to be detected of each reference 3D point to obtain the current spliced ​​features to be detected, that is, the query vector features of each reference 3D point.

[0171] Then, the electronic device inputs the current splicing features to be detected into the three-dimensional detection decoder module 304, and decodes the current splicing features to be detected by the decoder included in the three-dimensional detection decoder module 304 to obtain the current decoding result; based on the decoding result, the current coordinates and reference features of each reference three-dimensional point are updated. Then, the current coordinates and reference features of each reference three-dimensional point are sent to the learnable bird's-eye view feature extraction module 303 based on the query vector. Through the learnable bird's-eye view feature extraction module 303 based on the query vector, the current splicing features to be detected are obtained based on the current coordinates and reference features of each reference three-dimensional point, and then the three-dimensional detection decoder module 304 decodes the current splicing features to be detected. This cycle is repeated until the number of decoding times reaches the preset number, and the current coordinates of each reference three-dimensional point are obtained. Subsequently, the electronic device can obtain the position of the object in the spatial area to be detected based on the current coordinates of each reference three-dimensional point.

[0172] Based on the above processing, the electronic device can obtain the current fused point cloud features to be detected based on the point cloud to be detected; and obtain the current intermediate image features to be detected based on the image to be detected. Based on the current fused point cloud features to be detected and the current intermediate image features to be detected, the current coordinates and reference features of each reference three-dimensional point are updated. After a preset number of updates, the position of the object in the spatial area to be detected is determined according to the current coordinates of each reference three-dimensional point. That is, target detection is performed on the spatial area to be detected based on the point cloud and the image. In this way, since the process of acquiring the point cloud is not easily affected by external interference such as changes in lighting, changes in background color, and obstructions, combining the point cloud and the image to perform target detection together can make up for the defect of low accuracy of the target detection results obtained based only on the image, reduce the impact of external interference on the accuracy of the target detection results, and improve the accuracy of the target detection results obtained.

[0173] According to the target detection method provided in the embodiment of the present application, in the radar traffic scene, the electronic device can perform target detection of three-dimensional objects based on millimeter-wave radar and images (hereinafter referred to as radar-vision fusion three-dimensional detection). That is, in the digital intersection scene, the radar-vision fusion end-to-end model framework based on BEV's Query, based on the learnable RV matrix mapping sampling, realizes a more robust "BEV-image" mapping, making the radar-vision fusion three-dimensional detection more versatile and more generalizable. Detection through an end-to-end three-dimensional target detection model can reduce the customization and maintenance costs of the post-fusion logic, and can make full use of the information of multiple sensors.

[0174] Due to the differences in the performance of different sensors, such as the images collected by visual sensors contain rich semantic information, so that under normal circumstances, the accuracy of the detection results obtained based on visual sensors is relatively high, and they have been widely used in perception tasks such as target detection and target segmentation. However, when subject to external interference such as changes in illumination, changes in background color, and occlusions, as well as when the object's movement path is complex and the background color is messy, the accuracy of the detection results based on visual sensors will drop significantly, that is, the robustness of target detection based on visual sensors is not high. Radar sensors can measure information such as the speed, relative distance, and azimuth of an object, and are not easily affected by the above external interference, but the detection results obtained based on radar sensors are relatively sparse and noisy, that is, the robustness of target detection based on radar sensors is high, but the accuracy of the detection results is relatively low. Therefore, the all-weather perception capability of the algorithm can be improved by fusing multi-sensor information such as radar and vision, that is, the target detection method has high robustness and high accuracy.

[0175] Compared with the method of performing target detection based on visual information (i.e., image) and radar information (i.e., point cloud) respectively, and then performing radar-visual post-fusion in BEV space, since the post-fusion processing method will have problems such as insufficient use of sensor information, multiple logical rules, and high maintenance cost, the target detection method provided by this application, by fusing visual information and radar information, and then performing target detection based on the fused information, has obvious generalization performance advantages, and has better versatility and stronger generalization ability than the post-fusion processing method. It is compatible with a variety of sensors, such as pluggable support for multi-modal data such as radar, vision, and laser, and has extremely high multi-scene adaptability. In addition, compared with the post-fusion processing method under the image perspective, since the target scale is unified under the BEV space, the features under the BEV space can more intuitively represent the spatial position, size, orientation, distance and other information of the object, and can provide a unified physical space to realize the information fusion of different sensors. Therefore, fusion under the BEV space, that is, the subsequent detection of the position of the object in the BEV space, has better algorithm performance advantages than post-fusion under the image perspective.

[0176] Compared with the method of transforming images into pseudo point clouds based on display transformation, or processing images based on depth prediction to obtain BEV dense features, and then extracting local features of point clouds based on attention mechanisms, these methods have problems such as feature redundancy and the need for a large amount of invalid computational overhead. Based on the target detection method provided in this application, a learnable Query is used to represent the position of the object, image features are extracted based on the RV matrix mapping relationship, and features of different sensors are fused in the BEV space, thereby improving the overall 3D detection performance of the radar-vision fusion and the generalization of the algorithm. There is no need to map all the images to be detected into the BEV space, which can reduce invalid computational overhead.

[0177] The present application also provides a model training method, which is also applied to electronic devices. In order to distinguish it from the aforementioned electronic device for target detection, the electronic device used to execute the model training method is hereinafter referred to as the training device. The training device and the electronic device can be the same device, or they can be different devices, and the present application does not limit this. In the case where the training device and the electronic device are different devices, the training device can send the trained three-dimensional target detection model to the electronic device after executing the model training method provided by the present application to obtain a trained three-dimensional target detection model, so that the electronic device can perform target detection based on the deployed three-dimensional target detection model after deploying the received three-dimensional target detection model.

[0178] See also Figure 5 , Figure 5 A flowchart of a three-dimensional object detection model training method provided in an embodiment of the present application, the method may include the following steps:

[0179] S501: Acquire a point cloud and an image of a sample space region as a sample point cloud and a sample image, respectively, and acquire a sample label including a position of an object in the sample space region.

[0180] S502: Divide the sample point cloud according to the bird's-eye view grids, and determine the sample three-dimensional points contained in each bird's-eye view grid obtained by the division; use the point cloud feature extraction network in the three-dimensional target detection model of the initial structure to extract features of the sample three-dimensional points contained in each bird's-eye view grid to obtain the sample initial point cloud features of each bird's-eye view grid; and input the sample image into the image feature extraction network in the three-dimensional target detection model to obtain multiple sample initial image features.

[0181] Among them, the information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point, and the coordinates of the center of mass of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder.

[0182] S503: For each reference 3D point, the current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the bird's-eye view grid to which the offset coordinates belong is determined as the bird's-eye view grid corresponding to the position offset; the sample initial point cloud features of the bird's-eye view grids corresponding to each position offset are fused according to the weights corresponding to each position offset to obtain the current sample fused point cloud features of the reference 3D point.

[0183] S504: Input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each sample initial image feature; according to the weight of each sample initial image feature, fuse the corresponding feature values ​​of the reference 3D point in each sample initial image feature to obtain the current sample intermediate image feature of the reference 3D point.

[0184] S505: stitching the current sample fusion point cloud features and the sample intermediate image features of each reference three-dimensional point to obtain the current sample stitching features.

[0185] S506: Input the current sample splicing features into the decoder to obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the step of inputting the current reference features of the reference 3D point into the deformable attention network for each reference 3D point, and obtaining a preset number of sampling position offsets and a weight corresponding to each position offset, until the number of decoding times reaches the preset number.

[0186] S507: Obtain detection results based on the current coordinates of each reference three-dimensional point.

[0187] The detection result includes the position of the object in the sample space area.

[0188] S508: Based on the difference between the sample label and the detection result, the model parameters of the three-dimensional object detection model of the initial structure are adjusted until a preset convergence condition is reached to obtain a trained three-dimensional object detection model.

[0189] Based on the three-dimensional target detection model training method provided by the present application, the training device can obtain a trained three-dimensional target detection model, and then send the trained three-dimensional target detection model to the electronic device, so that the electronic device can perform target detection based on the deployed three-dimensional target detection model after deploying the received three-dimensional target detection model. When the electronic device performs target detection based on the end-to-end three-dimensional target detection model, it can combine point clouds and images to perform target detection together, thereby improving the accuracy of the target detection results, reducing maintenance costs, and improving the versatility of the target detection method.

[0190] For step S501, the sample space area and the categories of objects that may exist therein are related to the business requirements of the application scenario to which the trained 3D object detection model is to be applied. If the trained 3D object detection model needs to be applied in a digital intersection scenario, the sample space area may include roads around the traffic intersection, and the objects that may exist therein may include vehicles, roadblocks, etc.

[0191] By scanning the sample space area with a visual sensor, an image of the sample space area can be collected; by scanning the sample space area with a radar sensor, a point cloud of the sample space area can be collected. The training device can obtain an image of the sample space area from the visual sensor as a sample image, and obtain a point cloud of the sample space area from the radar sensor as a sample point cloud.

[0192] For step S502, the training device may divide the sample space area into multiple BEV grids, and one grid may be called a Pillar. The sample point cloud includes multiple three-dimensional points. For each three-dimensional point, the training device may determine the Pillar to which the three-dimensional point belongs, that is, determine the BEV grid to which the three-dimensional point belongs. In this way, the training device divides the sample point cloud according to the BEV grids and determines the three-dimensional points contained in each BEV grid.

[0193] The coordinates of the three-dimensional point can be the coordinates in the world coordinate system. When dividing the sample point cloud, for each three-dimensional point included in the sample point cloud, the training device can determine the area to which the three-dimensional point belongs only according to the coordinate value of the three-dimensional point on the X-axis and the coordinate value on the Y-axis. The BEV grid corresponding to the area is the BEV grid to which the three-dimensional point belongs. This can reduce the amount of calculation when dividing the sample point cloud, reduce the time consumption of division, and thus reduce the time consumption of model training and improve the efficiency of model training.

[0194] For each BEV grid, after determining the three-dimensional points included in the BEV grid, the training device can determine the minimum circumscribed cube of the three-dimensional points included in the BEV grid, and then use the centroid of the minimum circumscribed cube as the centroid of the BEV grid.

[0195] Since the format of the input data of the point cloud feature extraction network is set, and the number of three-dimensional points contained in each BEV grid may be different, in order to obtain the sample initial point cloud features of each BEV grid through the point cloud feature extraction network, the training device can determine the sample three-dimensional points contained in the BEV grid according to the number of three-dimensional points contained in each BEV grid.

[0196] Furthermore, for each BEV grid, the electronic device can input the sample three-dimensional points contained in the BEV grid into the point cloud feature extraction network in the three-dimensional target detection model of the initial structure, and perform feature extraction on the to-be-used information of the sample three-dimensional points contained in the BEV grid through the point cloud feature extraction network to obtain the sample initial point cloud features of the BEV grid.

[0197] In some embodiments, when the radar sensor can collect the coordinate value of the three-dimensional point on the Z axis, the coordinate value of the three-dimensional point on the Z axis contained in the to-be-used information of a three-dimensional point may be: the real coordinate value collected by the radar sensor. When the radar sensor fails to collect the coordinate value of the three-dimensional point on the Z axis, a preset virtual coordinate value may be used as the coordinate value of each three-dimensional point on the Z axis. For example, the preset virtual coordinate value may be set to 0 or 1.

[0198] After obtaining the sample image, the training device can use the image feature extraction network to extract features from the sample image to obtain multiple sample initial image features. For example, the image feature extraction network may include ResNet, VGG, etc. in the aforementioned embodiments, but is not limited thereto.

[0199] For step S503, for each reference 3D point, the training device can input the current reference feature of the reference 3D point into DAT, and obtain the preset number of position offsets that need to be obtained. Based on the deformable attention mechanism, DAT processes the current reference feature of the reference 3D point and can output the set number of position offsets and the weight corresponding to each position offset. For each position offset, the training device can use the position offset to offset the current coordinates of the reference 3D point.

[0200] In the case where the training device directly generates coordinates belonging to the sample space area as the coordinates of the reference three-dimensional point, the training device can directly calculate the sum of the current coordinates of the reference three-dimensional point and the position offset, and the calculated coordinates are the offset coordinates. In the case where the coordinates of the reference three-dimensional point are obtained by the training device based on a number randomly determined in the interval [0, 1], the training device can first scale the coordinates of the reference three-dimensional point according to the size of the sample space area, and then calculate the sum of the scaled coordinates and the position offset, and the calculated coordinates are the offset coordinates.

[0201] The BEV grid to which the offset coordinate belongs is the BEV grid corresponding to the position offset, that is, the BEV grid sampled based on the position offset. According to the weights corresponding to each position offset, the weighted sum of the sample initial point cloud features of the BEV grid corresponding to each position offset is calculated to obtain the current sample fusion point cloud features of the reference 3D point.

[0202] For step S504, for each reference 3D point, the training device may input the current reference feature of the reference 3D point into the weighted attention network. The weighted attention network processes the current reference feature of the reference 3D point based on the Attention mechanism, and may output the weights of the respective sample initial image features. After determining the corresponding part of the reference 3D point in the initial image features of each sample, that is, determining the corresponding feature value of the reference 3D point in the initial image features of each sample, the training device may calculate the weighted sum of the determined feature values ​​according to the weights of the initial image features of each sample, and obtain the current sample intermediate image feature of the reference 3D point.

[0203] For step S505 and step S506, after the training device splices the current sample fusion point cloud features and the sample intermediate image features of each reference three-dimensional point to obtain the current sample splicing features, the current sample splicing features can be input into the decoder to obtain the current decoding result. The decoder can be a Decoder based on a deep learning model in the aforementioned embodiment, such as a decoding module in a Transformer. The decoding result may include the position offset of each reference three-dimensional point, the reference feature offset of each reference three-dimensional point, etc. Based on the offset obtained by decoding, the training device can update the current coordinates, reference features, etc. of each reference three-dimensional point.

[0204] Then, the training device can use the current coordinates and reference features of each reference 3D point updated by this decoding to return to the step of inputting the current reference features of the reference 3D point into the deformable attention network for each reference 3D point, and obtaining a preset number of sampling position offsets and a weight corresponding to each position offset. That is, based on the current coordinates and reference features of each reference 3D point, the current sample splicing features are determined again, so that the next decoding is performed based on the current sample splicing features, and the coordinates and reference features of each reference 3D point are updated again until the number of decodings reaches the preset number. The preset number can be determined based on the type of decoder used.

[0205] For step S507 and step S508, after the number of decoding times reaches the preset number, for each reference 3D point, the current coordinates of the reference 3D point are the predicted positions of the objects existing in the sample space area, that is, the obtained detection results. Then the training device can obtain the difference between the sample label and the detection result. For example, the training device can calculate the loss function value representing the difference between the sample label and the detection result based on the preset loss function, and the preset loss function can be a cross entropy loss function, a mean square error loss function, etc.

[0206] In some embodiments, the decoding result may also include the confidence of the reference 3D point, that is, the probability that an object exists at the position indicated by the reference 3D point. When the confidence of a reference 3D point is greater than the confidence threshold, it indicates that the probability that an object exists at the reference 3D point is high, and the coordinates of the reference 3D point can be used as the position of the object in the most measured sample space area, as the target detection result for the sample space area. Subsequently, the training device can determine the difference between the detection result determined based on the confidence and the sample label.

[0207] Then, the training device can adjust the model parameters of the three-dimensional target detection model of the initial structure based on the obtained differences until the preset convergence conditions are reached to obtain a trained three-dimensional target detection model. That is, the parameters of the point cloud feature extraction network, image feature extraction network, deformable attention network, weighted attention network, and decoder included in the three-dimensional target detection model of the initial structure are adjusted. The preset convergence conditions may include any of the following: the number of training times reaches the preset number of training times, and the difference between the loss function value calculated this time and the loss function value calculated last time is less than the preset difference threshold.

[0208] In some embodiments, when a trained three-dimensional target detection model is obtained, the current coordinates and reference features of each reference three-dimensional point can be the coordinates and reference features of the reference three-dimensional point directly obtained by the electronic device in the aforementioned target detection method. Then, based on the aforementioned target detection method, the electronic device will subsequently adjust the coordinates and reference features of the reference three-dimensional point obtained, that is, adjust the coordinates and reference features of the reference three-dimensional point obtained during training, which can improve the accuracy of the position of the object in the detected spatial area compared to adjusting the coordinates and reference features of the reference three-dimensional point generated by the electronic device.

[0209] In some embodiments, determining the sample three-dimensional points contained in each BEV grid obtained by division may include the following steps: for each BEV grid obtained by division, if the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than a specified number, sampling a specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is equal to the specified number, then using the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, calculating the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid, and generating three-dimensional points whose coordinates are preset values; combining the generated three-dimensional points and the three-dimensional points in the sample point cloud contained in the BEV grid, the sample three-dimensional points contained in the BEV grid are obtained.

[0210] For each BEV grid obtained by division, when the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than the specified number, that is, the information to be used of the three-dimensional points contained in the BEV grid does not need to be fully input into the point cloud feature extraction network. At this time, the training device can sample the three-dimensional points contained in the BEV grid, and use the specified number of sampled three-dimensional points as the sample three-dimensional points contained in the BEV grid. Correspondingly, in the input data of the point cloud feature extraction network, the data corresponding to the BEV grid is the information to be used of the sample three-dimensional points.

[0211] When the number of 3D points in the sample point cloud contained in the BEV grid is equal to the specified number, the information to be used of all 3D points contained in the BEV grid is input into the point cloud feature extraction network. At this time, the training device can directly determine all 3D points contained in the BEV grid as sample 3D points. Correspondingly, in the input data of the point cloud feature extraction network, the data corresponding to the BEV grid is the information to be used of all 3D points in the BEV grid.

[0212] In the case where the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, even if all the three-dimensional points contained in the BEV grid are determined as sample three-dimensional points, the data corresponding to the BEV grid in the above-set data format cannot be obtained. At this time, the training device can perform supplementary processing on the data corresponding to the BEV grid. Specifically, the training device can calculate the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid, and generate a three-dimensional point whose coordinates are preset values ​​of the difference. When all values ​​in the information to be used of a three-dimensional point are preset values, the three-dimensional point can be indicated to be used to fill the input data so that the input data has a fixed format and is not a three-dimensional point contained in the point cloud of the sample space area. For example, the preset value can be 0 or 1.

[0213] Based on the above processing, when the number of three-dimensional points contained in a BEV grid is not the specified number, the training device can determine the sample three-dimensional points contained in the BEV grid based on the number of three-dimensional points contained in the BEV grid, based on sampling or supplementary processing. Subsequently, the point cloud feature extraction network can be used to extract features from the unused information of the sample three-dimensional points contained in each BEV grid, thereby reducing the probability of problems such as the low accuracy of the extracted sample initial point cloud features of each BEV grid due to the inconsistent number of three-dimensional points contained in each BEV grid, and thus improving the accuracy of the target detection results obtained based on the sample initial point cloud features of each BEV grid.

[0214] In some embodiments, the information to be utilized of a three-dimensional point also includes at least one of the following: the coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the real position corresponding to the detected three-dimensional point.

[0215] When the information to be used for a 3D point also includes the coordinate offset of the 3D point relative to the centroid of the BEV grid to which the 3D point belongs, that is, the input data of the point cloud feature extraction network already carries data indicating the relationship between some input data, the point cloud feature extraction network will have lower difficulty in learning the relationship between the input data, and the learned relationship between the input data will be more accurate. In this case, the sample initial point cloud features of each BEV grid will be more accurate.

[0216] When the information to be used of a three-dimensional point also includes: the feature data at the real position corresponding to the detected three-dimensional point, the point cloud feature extraction network performs feature extraction based on the feature data at each real position actually detected, and the extracted sample point cloud features of each BEV grid are more in line with the actual scene, that is, more accurate, and the subsequent target detection results based on the more accurate sample point cloud features of each BEV grid are more accurate. For example, the feature data may be signal-to-noise ratio, reflection intensity, etc. In the case where the radar sensor can detect multiple feature data at the real position corresponding to the three-dimensional point, the information to be used of the three-dimensional point may also include the multiple feature data to further improve the accuracy of the extracted sample initial point cloud features of each BEV grid.

[0217] In some embodiments, the image feature extraction network includes multiple downsampling layers connected in series; there is one sample image, and the multiple sample initial image features include: image features of the sample image extracted by the multiple downsampling layers; or, the sample image includes images of the sample space area captured at multiple different shooting angles, and the multiple sample initial image features include image features of each sample image.

[0218] After the sample image is input into the first downsampling layer in the serial connection, the multiple downsampling layers can perform downsampling processing on their own input data in turn. The output data of different downsampling layers, that is, the image features of the sample image extracted by the multiple downsampling layers, are different sample initial image features.

[0219] Therefore, when there is only one sample image, multiple sample initial image features can be extracted through the multiple serially connected downsampling layers included in the image feature extraction network.

[0220] When there are multiple sample images, that is, when multiple visual sensors collect images of the sample space area from different shooting angles, for each sample image, an image feature of the sample image can be extracted through an image feature extraction network; or, multiple sample initial image features of each sample image can be extracted separately in the manner of extracting multiple sample initial image features of a sample image as mentioned above.

[0221] Based on the above processing, multiple sample initial image features of the sample image are extracted through the image feature extraction network, that is, richer sample initial image features containing more information can be obtained. Then, the subsequent acquisition of prediction labels based on the sample initial image features containing more information is to obtain prediction labels by integrating more information, which can improve the accuracy of the acquired prediction labels.

[0222] In some embodiments, the three-dimensional object detection model also includes an MLP (Multilayer Perceptron).

[0223] Before the current reference feature of each reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and the weight corresponding to each position offset, the method may also include the following steps: Step 1: Input the sample initial point cloud features of each BEV grid and the sample initial image features into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix. The initial transformation matrix is ​​obtained by calibration according to the acquisition method of the sample point cloud and the sample image. Step 2: Use the obtained correction amount to correct the initial transformation matrix.

[0224] Correspondingly, the aforementioned steps fuse the eigenvalues ​​corresponding to the reference 3D point in each sample initial image feature according to the weight of each sample initial image feature to obtain the current sample intermediate image feature of the reference 3D point, which may include the following steps: Step 3: Use the corrected transformation matrix to transform the current coordinates of the reference 3D point to obtain the pixel coordinates corresponding to the reference 3D point in the sample image as the sample reference coordinates. Step 4: According to the weight of each sample initial image feature, fuse the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature to obtain the current sample intermediate image feature of the reference 3D point.

[0225] The initial transformation matrix can be obtained by calibrating according to the acquisition method of the sample point cloud and the sample image. The method of calibrating according to the acquisition method of the sample point cloud and the sample image is similar to the aforementioned method of calibrating according to the acquisition method of the point cloud to be detected and the image to be detected, and reference can be made to the relevant introduction of the aforementioned embodiment.

[0226] The training device can concat the sample initial point cloud features and sample initial image features of each BEV grid, and then input the concat features into the MLP to obtain a 3×3 matrix output by the MLP as a correction amount for correcting the initial transformation matrix.

[0227] In some embodiments, the aforementioned inputting of the sample initial point cloud features and the sample initial image features of each BEV grid into a multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix may include the following steps: inputting the sample initial point cloud features, the sample initial image features, and the current coordinates of each reference three-dimensional point of each BEV grid into a multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0228] It is understandable that at this time, decoding based on the reference 3D points has not been performed, and the coordinates of the reference 3D points have not been updated. When the training device generates the coordinates of the reference 3D points, the current coordinates of each reference 3D point are the coordinates generated by the training device; when the training device performs target detection on the sample space area based on the 3D target detection model, the current coordinates of each reference 3D point are the current coordinates of each reference 3D point after the 3D target detection model training is completed.

[0229] The training device can concat the sample initial point cloud features of each BEV grid, multiple sample initial image features, and the current coordinates of each reference three-dimensional point, and then input the concat features into the MLP, perform feature extraction on the concat features through the MLP, and obtain a 3×3 matrix output by the MLP as the correction amount for correcting the initial transformation matrix. In this way, the current coordinates of the reference three-dimensional points that comprehensively indicate the position of the object in the sample space area are used to jointly determine the correction amount, and the correction amount can indicate the position of the object in the sample space area. After the initial transformation matrix is ​​corrected based on the correction amount, the position of the existing object indicated by the sample reference position obtained based on the corrected transformation matrix is ​​more accurate, thereby improving the accuracy of the trained three-dimensional target detection model, so as to improve the accuracy of the subsequent target detection results obtained based on the three-dimensional target detection model.

[0230] Then, the training device can use the obtained correction amount to correct the initial transformation matrix. That is, for each element in the initial transformation matrix, calculate the sum of the element and the element at the same position of the element in the obtained correction amount, and update the element. The elements in the initial transformation matrix are updated, that is, the initial transformation matrix is ​​corrected using the obtained correction amount. Furthermore, for each reference three-dimensional point, the training device can use the corrected transformation matrix to transform the coordinates of the reference three-dimensional point to obtain the pixel coordinates in the pixel coordinate system, that is, obtain the corresponding sample reference coordinates in the sample image. The eigenvalues ​​corresponding to the sample reference coordinates in the features of each sample initial image, that is, the eigenvalues ​​corresponding to the reference three-dimensional point in the features of each sample initial image.

[0231] Based on the above processing, the training device can correct the initial transformation matrix obtained by calibration based on the sample initial point cloud features of each BEV grid and the sample initial image features. That is, the initial transformation matrix is ​​corrected based on the actual application scenario, and the corrected transformation matrix is ​​more in line with the actual application scenario than the initial transformation matrix. The current sample intermediate image features obtained based on the corrected transformation matrix are more accurate than the current sample intermediate image features obtained based on the initial transformation matrix. The subsequent three-dimensional target detection model trained based on the more accurate current sample intermediate image features has higher accuracy, thereby improving the accuracy of the subsequent target detection results obtained based on the three-dimensional target detection model.

[0232] In some embodiments, since the training device needs to generate a correction amount for correcting the initial transformation matrix based on the sample initial point cloud features of each BEV grid, that is, the sample initial point cloud features of each BEV grid also need to have a specified format. After obtaining the features output by the point cloud feature extraction network, the training device can also upgrade the output features to obtain a tensor with a dimension of (C, N, P); then pool the upgraded tensor of each BEV grid in the dimension of N to obtain a tensor with a dimension of (C, P). As mentioned above, P is the product of H and W, then the tensor with a dimension of (C, P) can be transformed into a tensor with a dimension of (C, H, W) as the sample initial point cloud features of each BEV grid. Among them, C is the number of output channels of the image feature extraction network.

[0233] For each reference 3D point, the sample reference coordinates obtained by performing coordinate transformation on the current coordinates of the reference 3D point may not be integers, that is, the sample reference coordinates may not correspond to the pixels in the sample image.

[0234] In some embodiments, when the sample reference coordinates are not integers, the training device can round the non-integer coordinate values ​​of the sample reference coordinates, and use the feature values ​​corresponding to the coordinates obtained after rounding in the initial image features of each sample as the feature values ​​corresponding to the sample reference coordinates.

[0235] In some embodiments, before the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature are fused according to the weights of each sample initial image feature to obtain the current sample intermediate image feature of the reference three-dimensional point, the method may further include the following steps: if the sample reference coordinate is not an integer, then other pixel coordinates within the neighborhood of the sample reference coordinate are obtained; for each sample initial image feature, the eigenvalues ​​corresponding to the other obtained pixel coordinates in the sample initial image feature are interpolated as the eigenvalues ​​corresponding to the sample reference coordinate in the sample initial image feature.

[0236] The training device can obtain other pixel coordinates within the neighborhood of the sample reference coordinates, such as rounding up and rounding down the coordinate values ​​that are not integers in the sample reference coordinates, and determining the pixel coordinates at the coordinate values ​​determined by rounding as other pixel coordinates within the neighborhood of the sample reference coordinates.

[0237] At this time, other pixel coordinates have corresponding feature values ​​in each sample initial image feature. For each sample initial image feature, the training device can interpolate the feature values ​​corresponding to the other pixel coordinates obtained in the sample initial image feature as the feature values ​​corresponding to the sample reference coordinates in the sample initial image feature.

[0238] Based on the above processing, when the sample reference coordinate is not an integer, the training device can still comprehensively consider the coordinates of other pixels within the neighborhood of the sample reference coordinate to jointly determine the eigenvalue corresponding to the sample reference coordinate. The eigenvalue corresponding to the sample reference coordinate determined in this way will be more accurate. The current sample intermediate image feature of the reference three-dimensional point subsequently calculated based on the more accurate eigenvalue will be more accurate, which can improve the accuracy of the target detection results subsequently obtained based on the current sample intermediate image feature.

[0239] In some embodiments, the sample label further includes: for each object in the sample space area, the category of the object; the detection result further includes: for each reference three-dimensional point, the category of the object at the position indicated by the reference three-dimensional point obtained based on the decoding result; the difference between the sample label and the detection result includes: the position difference between the position of the object contained in the sample label and the position of the object contained in the detection result; and the category difference between the category of the object contained in the sample label and the category of the object contained in the detection result.

[0240] For each object in the sample space region, the position of the object contained in the sample label may indicate: the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object.

[0241] Correspondingly, for each reference three-dimensional point, the detection result includes the position of the object at the position indicated by the reference three-dimensional point, which can indicate: the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object.

[0242] It is understandable that there may be multiple objects in the sample space area; there may be multiple reference three-dimensional points indicating the location of the object. Therefore, for each object in the sample space area, the position of the smallest circumscribed cube of the object (hereinafter referred to as the sample position) can be determined according to the position of the object contained in the sample label. Furthermore, the training device can calculate the intersection and union ratio of the sample position and the position indicated by each reference three-dimensional point (hereinafter referred to as the detection position), and determine the detection position with an intersection and union ratio greater than a preset intersection and union ratio threshold as the detection result corresponding to the object. Obviously, the detection result may be one or may include multiple results.

[0243] As mentioned above, the position of an object may include the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object; or, the position of an object may also include the coordinates of the vertices of the minimum circumscribed cube of the object.

[0244] Then, when the position of an object includes the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object, for each detection result corresponding to the object, the training device can calculate the size difference between the size of the minimum circumscribed cube of the object contained in the sample label and the size of the minimum circumscribed cube of the object contained in the detection result; calculate the coordinate difference between the coordinates of the center point of the minimum circumscribed cube of the object contained in the sample label and the coordinates of the center point of the minimum circumscribed cube of the object contained in the detection result; calculate the orientation difference between the orientation of the minimum circumscribed cube of the object contained in the sample label and the orientation of the minimum circumscribed cube of the object contained in the detection result. Size difference, coordinate difference, orientation difference, that is, the position difference between the object and the detection result. Subsequently, the training device can train the three-dimensional target detection model of the initial structure based on the position difference between each object and each of the respective detection results.

[0245] Then, in the case where the position of an object includes the coordinates of each vertex of the minimum circumscribed cube of the object, for each detection result corresponding to the object, according to the coordinates of each vertex of the minimum circumscribed cube of the object contained in the sample label (hereinafter referred to as sample vertices), and the coordinates of each vertex contained in the detection result (hereinafter referred to as detection vertices), the sample vertex and detection vertex (hereinafter referred to as vertex pair) indicating the same vertex of the minimum circumscribed cube of the object can be determined. Furthermore, for each vertex pair, the training device can calculate the difference between the coordinates of the sample vertex and the detection vertex included in the vertex pair, the difference of each vertex pair, that is, the position difference between the object and the detection result. Subsequently, the training device can train the three-dimensional target detection model of the initial structure based on the position difference between each object and each of its respective detection results.

[0246] Furthermore, when performing target detection based on machine vision, the target detection result obtained for each detected object can also indicate the category of the object. Accordingly, when training a 3D target detection model for target detection, for each object in the sample space area, the sample label can also include the category of the object; for each reference 3D point, the detection result also includes: the category of the object at the position indicated by the reference 3D point obtained based on the decoding result.

[0247] It is understandable that the category of an object can be represented in the form of a vector. For example, the first element in the vector represents the probability that the category of the object is category A; the second element in the vector represents the probability that the category of the object is category B. If the category of the object in the sample space area is category A, the sample label can include vector 1 (1, 0). If the detection result includes vector 2 (0.7, 0.3), the training device can calculate the loss function value representing the difference between vector 1 and vector 2 based on the loss function, that is, the category difference of the object in the sample space area between the sample label and the detection result is obtained, and then the model parameters of the three-dimensional target detection model are adjusted based on the loss function value.

[0248] Correspondingly, in the embodiment of the present application, for each reference 3D point, the decoding result also includes: the coordinate adjustment amount of the center point of the object (hereinafter referred to as the first object) existing at the reference 3D point, the size adjustment amount of the minimum circumscribed cube of the first object, the orientation adjustment amount of the minimum circumscribed cube of the first object, and the category adjustment amount of the first object. The training device can use the decoding result to update the current coordinates of the reference 3D point, the size of the minimum circumscribed cube of the first object, the orientation of the first object, the category of the first object, etc.

[0249] Based on the above processing, the three-dimensional object detection model is trained according to the position and category of the objects in the sample space area. The trained three-dimensional object detection model can also detect the category of the objects in the space area to be detected to meet business needs.

[0250] Based on the same inventive concept as the above target detection method, the present application also provides a target detection device. Figure 6 , Figure 6 A structural diagram of a target detection device provided in an embodiment of the present application. The device includes:

[0251] The first acquisition module 601 is used to acquire a point cloud and an image of a space area to be detected as a point cloud to be detected and an image to be detected respectively;

[0252] The first feature extraction module 602 is used to divide the point cloud to be detected according to the bird's-eye view BEV grid, extract the initial point cloud features to be detected of each BEV grid, and perform feature extraction on the image to be detected to obtain the initial image features to be detected;

[0253] The first point cloud feature fusion module 603 is used to sample the BEV grid within the neighborhood of each reference 3D point based on the current coordinates and reference features of the reference 3D point, and fuse the sampled initial point cloud features to be detected of the BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point;

[0254] The first image feature fusion module 604 is used to obtain the corresponding part of the reference 3D point in the initial image feature to be detected, and obtain the current intermediate image feature to be detected of the reference 3D point;

[0255] The first stitching module 605 is used to stitch the current fused point cloud features to be detected and the intermediate image features to be detected of each reference 3D point to obtain the current stitching features to be detected;

[0256] The first updating module 606 is used to decode the current splicing feature to be detected, obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the steps of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point for each reference 3D point, and fusing the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches the preset number;

[0257] The position acquisition module 607 is used to obtain the position of the object in the spatial area to be detected based on the current coordinates of each reference three-dimensional point.

[0258] Optionally, the first feature extraction module 602 is specifically used to divide the point cloud to be detected according to the bird's-eye view BEV grid, and determine the three-dimensional points to be detected contained in each BEV grid obtained by the division; use the point cloud feature extraction network in the pre-trained three-dimensional target detection model to extract features of the to-be-detected three-dimensional points contained in each BEV grid to obtain the initial point cloud features to be detected of each BEV grid; and input the image to be detected into the image feature extraction network in the three-dimensional target detection model to obtain multiple initial image features to be detected; wherein the to-be-detected information of a three-dimensional point includes: the coordinates of the three-dimensional point, and the coordinates of the center of mass of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model is: trained based on sample images, sample point clouds and sample labels representing the positions of objects in the sample space area; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder; the first point cloud feature fusion module 603 is specifically used to extract features of the reference three-dimensional point for each reference three-dimensional point. The current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; the initial point cloud features to be detected of the BEV grids corresponding to each position offset are fused according to the weights corresponding to each position offset to obtain the current fused point cloud features to be detected of the reference 3D point; the first image feature fusion module 604 is specifically used to input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each initial image feature to be detected; according to the weights of each initial image feature to be detected, the feature values ​​corresponding to the reference 3D point in each initial image feature to be detected are fused to obtain the current intermediate image feature to be detected of the reference 3D point; the first update module 606 is specifically used to input the current splicing feature to be detected into the decoder to obtain the current decoding result.

[0259] Optionally, the three-dimensional target detection model also includes a multi-layer perceptron; the device also includes: a first correction amount acquisition module, which is used to input the current reference feature of the reference three-dimensional point into the deformable attention network before the first point cloud feature fusion module 603 executes for each reference three-dimensional point, inputs the current reference feature of the reference three-dimensional point into the deformable attention network, obtains a preset number of sampling position offsets, and the weight corresponding to each position offset, and executes the input of the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is: calibrated according to the acquisition method of the point cloud to be detected and the image to be detected; the initial transformation matrix is ​​corrected using the obtained correction amount; the first image feature fusion module 604 is specifically used to use the corrected transformation matrix to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the image to be detected as the reference coordinates to be detected; according to the weights of each initial image feature to be detected, the eigenvalues ​​corresponding to the reference coordinates to be detected in each initial image feature to be detected are fused to obtain the current intermediate image feature to be detected of the reference three-dimensional point.

[0260] Optionally, the first correction amount acquisition module is specifically used to input the initial point cloud features to be detected of each BEV grid, the initial image features to be detected, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0261] Optionally, the device also includes: a first interpolation module, which is used to execute in the first image feature fusion module 604, according to the weight of each initial image feature to be detected, to fuse the eigenvalues ​​corresponding to the reference coordinate to be detected in each initial image feature to be detected, so as to obtain the current intermediate image feature to be detected of the reference three-dimensional point, and if the reference coordinate to be detected is not an integer, obtain other pixel coordinates within the neighborhood of the reference coordinate to be detected; for each initial image feature to be detected, interpolate the eigenvalues ​​corresponding to the other obtained pixel coordinates in the initial image feature to be detected as the eigenvalues ​​corresponding to the reference coordinate to be detected in the initial image feature to be detected.

[0262] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the image to be detected is one, and the multiple initial image features to be detected include: image features of the image to be detected extracted by multiple downsampling layers; or, the image to be detected includes images of the spatial area to be detected captured at multiple different shooting angles, and the multiple initial image features to be detected include image features of each image to be detected.

[0263] Optionally, the first feature extraction module 602 is specifically used to, for each BEV grid obtained by division, if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than a specified number, sample the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is equal to the specified number, use the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, calculate the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid, and generate three-dimensional points whose coordinates are preset values ​​of the difference; combine the generated three-dimensional points and the three-dimensional points in the point cloud to be detected contained in the BEV grid to obtain the three-dimensional points to be detected contained in the BEV grid.

[0264] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the detected real position corresponding to the three-dimensional point.

[0265] Based on the target detection device provided by the embodiment of the present application, the electronic device can obtain the current fused point cloud features to be detected based on the point cloud to be detected; and obtain the current intermediate image features to be detected based on the image to be detected. Based on the current fused point cloud features to be detected and the current intermediate image features to be detected, the current coordinates and reference features of each reference three-dimensional point are updated. After a preset number of updates, the position of the object in the space area to be detected is determined according to the current coordinates of each reference three-dimensional point. That is, target detection is performed on the space area to be detected based on the point cloud and the image. In this way, since the process of acquiring the point cloud is not easily affected by external interference such as changes in lighting, changes in background color, and obstructions, combining the point cloud and the image to perform target detection together can make up for the defect of low accuracy of the target detection results obtained based only on the image, reduce the impact of external interference on the accuracy of the target detection results, and improve the accuracy of the target detection results obtained.

[0266] In the technical solution of this application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing point clouds and images involved are all carried out with authorization. It should be noted that the 3D target detection model in this embodiment is not a 3D target detection model for a specific scene and cannot reflect the scene information of a specific scene. It should be noted that the point clouds and images in this embodiment are from public data sets.

[0267] Based on the same inventive concept as the above-mentioned 3D object detection model training method, the present application also provides a 3D object detection model training device. Figure 7 , Figure 7 A structural diagram of a three-dimensional object detection model training device provided in an embodiment of the present application. The device includes:

[0268] The second acquisition module 701 is used to acquire a point cloud and an image of a sample space region as a sample point cloud and a sample image, respectively, and to acquire a sample label including a position of an object in the sample space region;

[0269] The second feature extraction module 702 is used to divide the sample point cloud according to the bird's-eye view BEV grid, and determine the sample three-dimensional points contained in each BEV grid obtained by the division; use the point cloud feature extraction network in the three-dimensional target detection model of the initial structure to extract features from the to-be-used information of the sample three-dimensional points contained in each BEV grid to obtain the sample initial point cloud features of each BEV grid; and input the sample image into the image feature extraction network in the three-dimensional target detection model to obtain multiple sample initial image features;

[0270] The information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point and the coordinates of the centroid of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder;

[0271] The second point cloud feature fusion module 703 is used to input the current reference feature of each reference 3D point into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, use the position offset to offset the current coordinates of the reference 3D point, and determine the BEV grid to which the offset coordinates belong as the BEV grid corresponding to the position offset; fuse the sample initial point cloud features of the BEV grids corresponding to each position offset according to the weight corresponding to each position offset, to obtain the current sample fused point cloud features of the reference 3D point;

[0272] The second image feature fusion module 704 is used to input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each sample initial image feature; according to the weight of each sample initial image feature, the feature values ​​corresponding to the reference 3D point in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference 3D point;

[0273] The second stitching module 705 is used to stitch the current sample fusion point cloud features and the sample intermediate image features of each reference 3D point to obtain the current sample stitching features;

[0274] The second updating module 706 is used to input the current sample splicing feature into the decoder to obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the step of inputting the current reference feature of the reference 3D point into the deformable attention network for each reference 3D point, obtaining a preset number of sampling position offsets, and a weight corresponding to each position offset, until the number of decoding times reaches the preset number;

[0275] A detection result acquisition module 707, configured to obtain a detection result based on the current coordinates of each reference three-dimensional point; wherein the detection result includes the position of the object in the sample space area;

[0276] The training module 708 is used to adjust the model parameters of the three-dimensional object detection model of the initial structure based on the difference between the sample label and the detection result until a preset convergence condition is reached to obtain a trained three-dimensional object detection model.

[0277] Optionally, the three-dimensional object detection model further includes a multi-layer perceptron; and the device further includes:

[0278] The second correction amount acquisition module is used for inputting the current reference feature of each reference 3D point into the deformable attention network before the second point cloud feature fusion module 703 executes the input of the sample initial point cloud features of each BEV grid and the sample initial image features into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix for each reference 3D point, and obtaining the correction amount for correcting the initial transformation matrix; wherein the initial transformation matrix is ​​obtained by calibration according to the acquisition method of the sample point cloud and the sample image; and the initial transformation matrix is ​​corrected by using the obtained correction amount; the second image feature fusion module 704 is specifically used for: using the corrected transformation matrix to transform the current coordinates of the reference 3D point to obtain the pixel coordinates corresponding to the reference 3D point in the sample image as the sample reference coordinates; according to the weights of the initial image features of each sample, the eigenvalues ​​corresponding to the sample reference coordinates in the initial image features of each sample are fused to obtain the current sample intermediate image features of the reference 3D point.

[0279] Optionally, the second correction amount acquisition module is specifically used to input the sample initial point cloud features of each BEV grid, the sample initial image features, and the current coordinates of each reference three-dimensional point into the multi-layer perceptron to obtain the correction amount for correcting the initial transformation matrix.

[0280] Optionally, the device also includes: a second interpolation module, which is used to execute, in the second image feature fusion module 704, fusing the eigenvalues ​​corresponding to the sample reference coordinates in each sample initial image feature according to the weights of each sample initial image feature, to obtain the current sample intermediate image feature of the reference three-dimensional point, and if the sample reference coordinate is not an integer, then obtain other pixel coordinates within the neighborhood of the sample reference coordinate; for each sample initial image feature, interpolate the eigenvalues ​​corresponding to the other obtained pixel coordinates in the sample initial image feature as the eigenvalues ​​corresponding to the sample reference coordinate in the sample initial image feature.

[0281] Optionally, the image feature extraction network includes multiple downsampling layers connected in series; the sample image is one, and the multiple sample initial image features include: image features of the sample image extracted by multiple downsampling layers; or, the sample image includes images of the sample space area captured at multiple different shooting angles, and the multiple sample initial image features include image features of each sample image.

[0282] Optionally, the second feature extraction module 702 is specifically used to, for each BEV grid obtained by division, if the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than a specified number, sample the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is equal to the specified number, use the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; if the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, calculate the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid, and generate three-dimensional points whose coordinates are preset values; combine the generated three-dimensional points and the three-dimensional points in the sample point cloud contained in the BEV grid to obtain the sample three-dimensional points contained in the BEV grid.

[0283] Optionally, the information to be used of a three-dimensional point also includes at least one of the following: a coordinate offset of the three-dimensional point relative to the center of mass of the BEV grid to which the three-dimensional point belongs, and feature data at the real position corresponding to the detected three-dimensional point.

[0284] Optionally, the sample label also includes the category of the object in the sample space area; the position indication of the object in the sample space area includes: the size of the minimum circumscribed cube of the object, the coordinates of the center point of the minimum circumscribed cube of the object, and the orientation of the object; the detection result also includes the category of the object in the sample space area; the difference between the sample label and the detection result includes: the difference in the position of the object in the sample space area, and the difference in the category of the object in the sample space area.

[0285] Based on the three-dimensional target detection model training device provided by the present application, the training device can obtain a trained three-dimensional target detection model, and then send the trained three-dimensional target detection model to the electronic device, so that the electronic device can perform target detection based on the deployed three-dimensional target detection model after deploying the received three-dimensional target detection model. When the electronic device performs target detection based on the end-to-end three-dimensional target detection model, it can combine point clouds and images to perform target detection together, thereby improving the accuracy of the target detection results, reducing maintenance costs, and improving the versatility of the target detection method.

[0286] The present application also provides an electronic device, such as Figure 8 As shown, it includes: a memory 801 for storing computer programs; a processor 802, for implementing the steps of any target detection method in the above embodiments when executing the program stored in the memory 801, or implementing the steps of any three-dimensional target detection model training method in the above embodiments.

[0287] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0288] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0289] The communication interface is used for communication between the above electronic device and other devices.

[0290] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0291] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0292] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned target detection methods, or implements the steps of any of the above-mentioned three-dimensional target detection model training methods.

[0293] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any target detection method in the above embodiments, or execute any three-dimensional target detection model training method in the above embodiments.

[0294] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0295] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0296] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0297] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Obtaining a point cloud and an image of the space area to be detected as the point cloud to be detected and the image to be detected respectively; Dividing the point cloud to be detected according to the bird's-eye view BEV grid, extracting the initial point cloud features to be detected of each BEV grid, and performing feature extraction on the image to be detected to obtain the initial image features to be detected; For each reference 3D point, based on the current coordinates and reference features of the reference 3D point, the BEV grid in the neighborhood of the reference 3D point is sampled, and the initial point cloud features to be detected of the sampled BEV grid are fused to obtain the current fused point cloud features to be detected of the reference 3D point; Acquire a portion of the reference three-dimensional point corresponding to the initial image feature to be detected, and obtain a current intermediate image feature to be detected of the reference three-dimensional point; The current fused point cloud features to be detected and the intermediate image features to be detected of each reference 3D point are spliced ​​to obtain the current spliced ​​features to be detected; Decode the current splicing feature to be detected to obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the steps of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point for each reference 3D point, and fusing the sampled initial point cloud features to be detected of the BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches the preset number; Based on the current coordinates of each reference three-dimensional point, the position of the object in the spatial area to be detected is obtained.

2. The method according to claim 1, characterized in that The step of dividing the point cloud to be detected according to the bird's-eye view BEV grid, extracting the initial point cloud features to be detected of each BEV grid, and performing feature extraction on the image to be detected to obtain the initial image features to be detected includes: The point cloud to be detected is divided according to the bird's-eye view BEV grid, and the three-dimensional points to be detected contained in each BEV grid obtained by the division are determined; the feature extraction of the information to be used of the three-dimensional points to be detected contained in each BEV grid is performed using the point cloud feature extraction network in the pre-trained three-dimensional target detection model to obtain the initial point cloud features to be detected of each BEV grid; and the image to be detected is input into the image feature extraction network in the three-dimensional target detection model to obtain a plurality of initial image features to be detected; The information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point and the coordinates of the centroid of the BEV grid to which the three-dimensional point belongs; the three-dimensional object detection model is obtained by training based on a sample image of a sample space region, a sample point cloud and a sample label representing the position of an object in the sample space region; the three-dimensional object detection model also includes a deformable attention network, a weighted attention network and a decoder; For each reference 3D point, based on the current coordinates and reference features of the reference 3D point, the BEV grid in the neighborhood of the reference 3D point is sampled, and the initial point cloud features to be detected of the sampled BEV grid are fused to obtain the current fused point cloud features to be detected of the reference 3D point, including: For each reference 3D point, the current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; the initial point cloud features to be detected of the BEV grids corresponding to each position offset are fused according to the weights corresponding to each position offset to obtain the current fused point cloud features to be detected of the reference 3D point; The obtaining of the portion of the reference 3D point corresponding to the initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference 3D point includes: inputting the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each initial image feature to be detected; fusing the feature values ​​corresponding to the reference 3D point in each initial image feature to be detected according to the weight of each initial image feature to be detected to obtain the current intermediate image feature to be detected of the reference 3D point; The decoding of the current splicing feature to be detected to obtain the current decoding result includes: inputting the current splicing feature to be detected into the decoder to obtain the current decoding result.

3. The method according to claim 2, characterized in that The three-dimensional target detection model also includes a multi-layer perceptron; Before inputting the current reference feature of each reference 3D point into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset, the method further includes: Input the initial point cloud features to be detected of each BEV grid and the initial image features to be detected into the multi-layer perceptron to obtain a correction amount for correcting the initial conversion matrix; wherein the initial conversion matrix is ​​obtained by calibration according to the acquisition method of the point cloud to be detected and the image to be detected; Correcting the initial transformation matrix using the obtained correction amount; According to the weights of the initial image features to be detected, the feature values ​​corresponding to the reference three-dimensional point in the initial image features to be detected are fused to obtain the current intermediate image feature to be detected of the reference three-dimensional point, including: The corrected transformation matrix is ​​used to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the image to be detected as the reference coordinates to be detected; According to the weight of each initial image feature to be detected, the feature values ​​corresponding to the reference coordinate to be detected in each initial image feature to be detected are fused to obtain the current intermediate image feature to be detected of the reference three-dimensional point.

4. The method according to claim 3, characterized in that The initial point cloud features to be detected of each BEV grid and the initial image features to be detected are input into the multi-layer perceptron to obtain a correction amount for correcting the initial conversion matrix, including: The initial point cloud features to be detected of each BEV grid, the initial image features to be detected, and the current coordinates of each reference three-dimensional point are input into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix.

5. The method according to claim 3, characterized in that: Before fusing the feature values ​​corresponding to the reference coordinate to be detected in each of the initial image features to be detected according to the weights of each of the initial image features to be detected to obtain the current intermediate image feature to be detected of the reference three-dimensional point, the method further includes: If the reference coordinate to be detected is not an integer, obtaining coordinates of other pixels within the neighborhood of the reference coordinate to be detected; For each initial image feature to be detected, feature values ​​corresponding to other acquired pixel coordinates in the initial image feature to be detected are interpolated as feature values ​​corresponding to the reference coordinates to be detected in the initial image feature to be detected.

6. The method according to claim 2, characterized in that The image feature extraction network includes a plurality of downsampling layers connected in series; the image to be detected is one, and the plurality of initial image features to be detected include: image features of the image to be detected extracted by the plurality of downsampling layers; or, The image to be detected includes images of the spatial area to be detected collected at multiple different shooting angles, and the multiple initial image features to be detected include image features of each image to be detected.

7. The method according to claim 2, characterized in that The three-dimensional points to be detected contained in each BEV grid obtained by the determination of the division include: For each BEV grid obtained by division, if the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is greater than a specified number, sampling the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the three-dimensional points to be detected contained in the BEV grid; If the number of the three-dimensional points in the to-be-detected point cloud contained in the BEV grid is equal to the specified number, the three-dimensional points contained in the BEV grid are used as the to-be-detected three-dimensional points contained in the BEV grid; If the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is less than the specified number, the difference between the specified number and the number of three-dimensional points in the point cloud to be detected contained in the BEV grid is calculated to generate three-dimensional points whose coordinates are preset values ​​of the difference; the generated three-dimensional points are combined with the three-dimensional points in the point cloud to be detected contained in the BEV grid to obtain the three-dimensional points to be detected contained in the BEV grid.

8. A three-dimensional object detection model training method, characterized in that: The method comprises: Acquire a point cloud and an image of a sample space region as a sample point cloud and a sample image, respectively, and acquire a sample label including a position of an object in the sample space region; The sample point cloud is divided according to the bird's-eye view BEV grid, and the sample three-dimensional points contained in each BEV grid obtained by the division are determined; the feature extraction of the to-be-used information of the sample three-dimensional points contained in each BEV grid is performed using the point cloud feature extraction network in the three-dimensional target detection model of the initial structure to obtain the sample initial point cloud features of each BEV grid; and the sample image is input into the image feature extraction network in the three-dimensional target detection model to obtain a plurality of sample initial image features; The information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point and the coordinates of the centroid of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder; For each reference 3D point, the current reference feature of the reference 3D point is input into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, the current coordinates of the reference 3D point are offset by using the position offset, and the BEV grid to which the offset coordinates belong is determined as the BEV grid corresponding to the position offset; the sample initial point cloud features of the BEV grids corresponding to each position offset are fused according to the weight corresponding to each position offset to obtain the current sample fused point cloud features of the reference 3D point; Inputting the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each sample initial image feature; fusing the feature values ​​corresponding to the reference 3D point in each sample initial image feature according to the weight of each sample initial image feature to obtain the current sample intermediate image feature of the reference 3D point; The current sample fusion point cloud features and the sample intermediate image features of each reference 3D point are spliced ​​to obtain the current sample splicing features; Input the current sample splicing feature into the decoder to obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the step of inputting the current reference features of the reference 3D point into the deformable attention network for each reference 3D point, obtaining a preset number of sampling position offsets, and a weight corresponding to each position offset, until the number of decoding times reaches the preset number; Based on the current coordinates of each reference three-dimensional point, a detection result is obtained; wherein the detection result includes the position of the object in the sample space area; Based on the difference between the sample label and the detection result, the model parameters of the three-dimensional object detection model of the initial structure are adjusted until a preset convergence condition is reached to obtain a trained three-dimensional object detection model.

9. The method according to claim 8, characterized in that The three-dimensional target detection model also includes a multi-layer perceptron; Before inputting the current reference feature of each reference 3D point into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset, the method further includes: Inputting the sample initial point cloud features of each BEV grid and the sample initial image features into the multi-layer perceptron to obtain a correction amount for correcting the initial conversion matrix; wherein the initial conversion matrix is ​​obtained by calibration according to the acquisition method of the sample point cloud and the sample image; Correcting the initial transformation matrix using the obtained correction amount; According to the weight of each sample initial image feature, the feature values ​​corresponding to the reference three-dimensional point in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference three-dimensional point, including: The corrected transformation matrix is ​​used to transform the current coordinates of the reference three-dimensional point to obtain the pixel coordinates corresponding to the reference three-dimensional point in the sample image as the sample reference coordinates; According to the weight of each sample initial image feature, the feature values ​​corresponding to the sample reference coordinates in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference three-dimensional point.

10. The method according to claim 9, characterized in that The sample initial point cloud features of each BEV grid and the sample initial image features are input into the multi-layer perceptron to obtain a correction amount for correcting the initial conversion matrix, including: The sample initial point cloud features of each BEV grid, the sample initial image features, and the current coordinates of each reference three-dimensional point are input into the multi-layer perceptron to obtain a correction amount for correcting the initial transformation matrix.

11. The method according to claim 9, characterized in that Before fusing the feature values ​​corresponding to the sample reference coordinates in the sample initial image features according to the weights of the sample initial image features to obtain the current sample intermediate image feature of the reference three-dimensional point, the method further includes: If the sample reference coordinate is not an integer, obtaining coordinates of other pixels within the neighborhood of the sample reference coordinate; For each sample initial image feature, feature values ​​corresponding to other acquired pixel coordinates in the sample initial image feature are interpolated as feature values ​​corresponding to the sample reference coordinates in the sample initial image feature.

12. The method according to claim 8, characterized in that The image feature extraction network includes a plurality of downsampling layers connected in series; the sample image is one, and the plurality of sample initial image features include: image features of the sample image extracted by the plurality of downsampling layers; or, The sample image includes images of the sample space area collected at multiple different shooting angles, and the multiple sample initial image features include image features of each sample image.

13. The method according to claim 8, characterized in that The sample three-dimensional points contained in each BEV grid obtained by the determination of the division include: For each BEV grid obtained by division, if the number of three-dimensional points in the sample point cloud contained in the BEV grid is greater than a specified number, sampling the specified number of three-dimensional points from the three-dimensional points contained in the BEV grid as the sample three-dimensional points contained in the BEV grid; If the number of three-dimensional points in the sample point cloud contained in the BEV grid is equal to the specified number, the three-dimensional points contained in the BEV grid are used as the sample three-dimensional points contained in the BEV grid; If the number of three-dimensional points in the sample point cloud contained in the BEV grid is less than the specified number, the difference between the specified number and the number of three-dimensional points in the sample point cloud contained in the BEV grid is calculated to generate three-dimensional points whose coordinates are preset values. The generated three-dimensional points are combined with the three-dimensional points in the sample point cloud contained in the BEV grid to obtain the sample three-dimensional points contained in the BEV grid.

14. A target detection device, characterized in that: The device comprises: A first acquisition module is used to acquire a point cloud and an image of a spatial area to be detected as a point cloud to be detected and an image to be detected, respectively; A first feature extraction module is used to divide the point cloud to be detected according to the bird's-eye view BEV grid, extract the initial point cloud features to be detected of each BEV grid, and perform feature extraction on the image to be detected to obtain the initial image features to be detected; The first point cloud feature fusion module is used to sample the BEV grid within the neighborhood of each reference 3D point based on the current coordinates and reference features of the reference 3D point, and fuse the initial point cloud features to be detected of the sampled BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point; A first image feature fusion module is used to obtain a portion of the reference three-dimensional point corresponding to the initial image feature to be detected, and obtain a current intermediate image feature to be detected of the reference three-dimensional point; A first stitching module is used to stitch the current fused point cloud features to be detected and the intermediate image features to be detected of each reference three-dimensional point to obtain the current stitching features to be detected; A first updating module is used to decode the current splicing feature to be detected, obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the steps of sampling the BEV grid within the neighborhood of the reference 3D point based on the current coordinates and reference features of the reference 3D point for each reference 3D point, and fusing the sampled initial point cloud features to be detected of the BEV grid to obtain the current fused point cloud features to be detected of the reference 3D point, until the number of decoding times reaches a preset number; The position acquisition module is used to obtain the position of the object in the spatial area to be detected based on the current coordinates of each reference three-dimensional point.

15. A three-dimensional object detection model training device, characterized in that: The device comprises: A second acquisition module is used to acquire a point cloud and an image of a sample space region as a sample point cloud and a sample image, respectively, and to acquire a sample label including a position of an object in the sample space region; The second feature extraction module is used to divide the sample point cloud according to the bird's-eye view BEV grid, and determine the sample three-dimensional points contained in each BEV grid obtained by the division; use the point cloud feature extraction network in the three-dimensional target detection model of the initial structure to extract features from the to-be-used information of the sample three-dimensional points contained in each BEV grid to obtain the sample initial point cloud features of each BEV grid; and input the sample image into the image feature extraction network in the three-dimensional target detection model to obtain multiple sample initial image features; The information to be used of a three-dimensional point includes: the coordinates of the three-dimensional point and the coordinates of the centroid of the BEV grid to which the three-dimensional point belongs; the three-dimensional target detection model also includes a deformable attention network, a weighted attention network and a decoder; The second point cloud feature fusion module is used to input the current reference feature of each reference 3D point into the deformable attention network to obtain a preset number of sampling position offsets and a weight corresponding to each position offset; for each position offset, use the position offset to offset the current coordinates of the reference 3D point, and determine the BEV grid to which the offset coordinates belong as the BEV grid corresponding to the position offset; fuse the sample initial point cloud features of the BEV grids corresponding to each position offset according to the weight corresponding to each position offset, to obtain the current sample fused point cloud features of the reference 3D point; The second image feature fusion module is used to input the current reference feature of the reference 3D point into the weighted attention network to obtain the weight of each sample initial image feature; according to the weight of each sample initial image feature, the feature values ​​corresponding to the reference 3D point in each sample initial image feature are fused to obtain the current sample intermediate image feature of the reference 3D point; The second stitching module is used to stitch the current sample fusion point cloud features and the sample intermediate image features of each reference three-dimensional point to obtain the current sample stitching features; The second updating module is used to input the current sample splicing feature into the decoder to obtain the current decoding result, and update the current coordinates and reference features of the reference 3D point based on the current decoding result; return to execute the step of inputting the current reference feature of the reference 3D point into the deformable attention network for each reference 3D point, and obtaining a preset number of sampling position offsets and a weight corresponding to each position offset, until the number of decoding times reaches the preset number; A detection result acquisition module, used to obtain a detection result based on the current coordinates of each reference three-dimensional point; wherein the detection result includes the position of the object in the sample space area; The training module is used to adjust the model parameters of the three-dimensional object detection model of the initial structure based on the difference between the sample label and the detection result until a preset convergence condition is reached to obtain a trained three-dimensional object detection model.

Images