Three-dimensional target detection method, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional target detection method, electronic equipment and a storage medium. The method comprises the following steps: determining at least one initial two-dimensional image feature corresponding to each two-dimensional image and a three-dimensional suggestion box corresponding to each three-dimensional point cloud data; dividing each three-dimensional suggestion box, determining an initial local grid feature of each three-dimensional suggestion box according to a division result, and determining a target local grid feature of each three-dimensional suggestion box based on each initial two-dimensional image feature and each initial local grid feature; determining a global grid feature corresponding to each three-dimensional suggestion box; and according to each initial local grid feature, each target local grid feature and each global grid feature, obtaining an aggregation feature of the three-dimensional target, and according to the aggregation feature, detecting the three-dimensional target. According to the method, the aggregation features of the three-dimensional target are obtained, and the features extracted according to different modal data are fully utilized, so that the accuracy and robustness of three-dimensional target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a three-dimensional target detection method, electronic equipment and storage medium. Background Art

[0002] In recent years, with the continuous development of the global economy and technology, more and more manufacturing companies have turned their attention to automated and unmanned production. In this highly automated production environment, the precise and rapid detection of people, machines, and objects in the manufacturing workshop is directly related to the operating efficiency of the entire production line and the quality level of the product. Therefore, it is crucial to ensure the accurate detection of people, machines, and objects in the manufacturing scene.

[0003] In the prior art, multimodal fusion technology is used to perform three-dimensional target detection by fusing data from different sensors. However, traditional multimodal fusion technology generally uses the voxel center to represent the position of each voxel feature, which will inevitably ignore the actual distribution of points within each voxel, resulting in a decrease in the accuracy of three-dimensional target detection. Summary of the invention

[0004] The purpose of this application is to provide a three-dimensional target detection method, electronic device and storage medium to improve the accuracy of three-dimensional target detection in order to address the deficiencies in the above-mentioned prior art.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a three-dimensional object detection method, the method comprising:

[0007] Acquire each two-dimensional image of the three-dimensional target and three-dimensional point cloud data corresponding to each of the two-dimensional images, and determine at least one non-empty voxel and a non-empty voxel feature corresponding to the non-empty voxel according to each of the three-dimensional point cloud data;

[0008] Input each of the two-dimensional images and each of the three-dimensional point cloud data into a feature extraction module of a pre-trained multimodal detection model, respectively, to obtain at least one initial two-dimensional image feature corresponding to each of the two-dimensional images and a three-dimensional suggestion box corresponding to each of the three-dimensional point cloud data, wherein the three-dimensional suggestion box is used to indicate a three-dimensional area where the three-dimensional object exists;

[0009] Inputting each of the three-dimensional suggestion boxes and each of the initial two-dimensional image features into the learnable function local fusion module of the pre-trained multimodal detection model, dividing each of the three-dimensional suggestion boxes and determining the initial local grid features of each of the three-dimensional suggestion boxes according to the division results, and determining the target local grid features of each of the three-dimensional suggestion boxes based on each of the initial two-dimensional image features and each of the initial local grid features;

[0010] Inputting each non-empty voxel corresponding to each of the three-dimensional suggestion boxes, each non-empty voxel feature, and each target two-dimensional image feature corresponding to each two-dimensional image into the point cloud image global fusion module of the pre-trained multimodal detection model to perform global grid feature processing, so as to obtain the global grid feature corresponding to each of the three-dimensional suggestion boxes, wherein the target two-dimensional image feature is obtained by performing semantic feature aggregation on the initial two-dimensional image feature;

[0011] According to each of the initial local grid features, each of the target local grid features and each of the global grid features, an aggregate feature of the three-dimensional target is obtained, and the three-dimensional target is detected according to the aggregate feature.

[0012] Optionally, dividing each of the three-dimensional suggestion boxes and determining an initial local grid feature of each of the three-dimensional suggestion boxes according to a division result, and determining a target local grid feature of each of the three-dimensional suggestion boxes based on each of the initial two-dimensional image features and each of the initial local grid features, includes:

[0013] Dividing each of the three-dimensional suggestion boxes into a plurality of voxel grids, extracting local grid features according to each of the voxel grids, and obtaining the initial local grid features of each of the three-dimensional suggestion boxes;

[0014] Obtaining, according to the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes and the initial local grid features of each of the three-dimensional suggestion boxes, the image-enhanced local grid features corresponding to each of the three-dimensional suggestion boxes;

[0015] The enhanced local grid features of each image and the initial local grid features are fused to obtain target local grid features of each three-dimensional suggestion box.

[0016] Optionally, dividing each of the three-dimensional suggestion boxes into a plurality of voxel grids, and extracting local grid features according to each voxel grid to obtain the initial local grid features includes:

[0017] Dividing each of the three-dimensional suggestion boxes to obtain a plurality of voxel grids corresponding to each of the three-dimensional suggestion boxes;

[0018] According to the variant neural network formula and each of the voxel grids, grid features of each of the voxel grids corresponding to each of the three-dimensional suggestion boxes are obtained;

[0019] A set of grid features of all voxel grids corresponding to each of the three-dimensional suggestion boxes is used as the initial local grid feature of each of the three-dimensional suggestion boxes.

[0020] Optionally, obtaining the image-enhanced local grid features of each of the three-dimensional suggestion boxes according to the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes and the initial local grid features of each of the three-dimensional suggestion boxes includes:

[0021] fusing the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes with the initial local grid features of each of the three-dimensional suggestion boxes to obtain fused local grid features of each of the three-dimensional suggestion boxes;

[0022] Generating first query features corresponding to each of the three-dimensional suggestion boxes according to each of the fused local grid features and each of the initial local grid features with position coding information;

[0023] Obtaining first sampling features corresponding to each of the two-dimensional images according to each of the target two-dimensional image features of the two-dimensional images corresponding to each of the three-dimensional suggestion boxes and sampling points corresponding to each of the three-dimensional suggestion boxes;

[0024] The image-enhanced local grid features of each of the three-dimensional suggestion boxes are obtained according to each of the first query features, each of the first sampling features, a reference point of the centroid of each voxel grid in each of the three-dimensional suggestion boxes on the two-dimensional image corresponding to the three-dimensional suggestion box, the total number of sampling points in each of the three-dimensional suggestion boxes, the total number of self-attention heads, the learnable weights, the first sampling offset and the first attention weight value.

[0025] Optionally, the inputting each non-empty voxel, each non-empty voxel feature and each target two-dimensional image feature corresponding to each of the three-dimensional suggestion boxes into a point cloud image global fusion module of the multimodal detection model for global grid feature processing to obtain the global grid feature corresponding to each of the three-dimensional suggestion boxes includes:

[0026] Determine, according to the 3D point cloud data in each non-empty voxel corresponding to each of the 3D suggestion boxes and the camera projection matrix, the point centroid of each of the non-empty voxels and the target point of the point centroid in a target 2D image, wherein the target 2D image is a 2D image corresponding to each of the 3D suggestion boxes;

[0027] Obtaining, according to each target point in each target two-dimensional image, each target two-dimensional image feature corresponding to each target two-dimensional image, each non-empty voxel feature corresponding to each three-dimensional suggestion box, a learnable weight, a second sampling offset, and a second attention weight value, an image-enhanced voxel feature corresponding to each three-dimensional suggestion box;

[0028] According to the voxel features after each image enhancement and the non-empty voxel features corresponding to each three-dimensional suggestion box, the global grid features corresponding to each three-dimensional suggestion box are obtained.

[0029] Optionally, determining the point centroid of each non-empty voxel and the target point of the point centroid in the target two-dimensional image according to the three-dimensional point cloud data in each non-empty voxel corresponding to the three-dimensional suggestion box and the camera projection matrix includes:

[0030] Taking the average value of the spatial coordinates of all the three-dimensional point cloud data in each of the non-empty voxels as the point centroid of the non-empty voxel;

[0031] According to the centroids of the points and the camera projection matrix, the target points of the centroids of the points in the target two-dimensional image are obtained.

[0032] Optionally, obtaining the image-enhanced voxel features corresponding to each of the three-dimensional suggestion boxes according to each of the target points in each of the target two-dimensional images, each of the target two-dimensional image features corresponding to the target two-dimensional images, each of the non-empty voxel features corresponding to each of the three-dimensional suggestion boxes, a learnable weight, a second sampling offset value, and a second attention weight value includes:

[0033] Performing weighted processing on each target two-dimensional image feature of each target point preset area in each target two-dimensional image to obtain an aggregated image feature corresponding to each target two-dimensional image;

[0034] According to the aggregated image features corresponding to each of the target two-dimensional images, each target point on each of the target two-dimensional images, the total number of point centroids in each of the three-dimensional suggestion boxes, each non-empty voxel feature corresponding to each of the three-dimensional suggestion boxes, the second sampling offset and the second attention weight value, the image-enhanced voxel features corresponding to each of the three-dimensional suggestion boxes are determined.

[0035] Optionally, obtaining the aggregated features of the three-dimensional target according to the initial local grid features, the target local grid features and the global grid features includes:

[0036] The initial local grid features, the target local grid features and the global grid features are weightedly added to obtain the aggregated features.

[0037] In a second aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when an application is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the three-dimensional target detection method described in the first aspect above.

[0038] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is read and executes the steps of the three-dimensional target detection method described in the first aspect above.

[0039] The beneficial effects of this application are:

[0040] The present application provides a three-dimensional target detection method, electronic device and storage medium. By inputting each three-dimensional suggestion box and each initial two-dimensional image feature into a learnable function local fusion module of a pre-trained multimodal detection model, each three-dimensional suggestion box is divided and the initial local grid feature of each three-dimensional suggestion box is determined according to the division result, so that the obtained initial local grid feature granularity can be finer and more accurate, and the subtle features of the three-dimensional target are extracted. The target local grid feature of each three-dimensional suggestion box is determined based on each initial two-dimensional image feature and each initial local grid feature. The target local grid feature of the three-dimensional suggestion box combines the features extracted from different modal data, so that the obtained target local grid feature is more accurate. At the same time, the point cloud image global fusion module is used to perform global grid feature processing on the point cloud data and the two-dimensional image to obtain the global grid feature corresponding to each three-dimensional suggestion box. Through the processing of the two fusion modules, the features extracted from the two-dimensional image and the features extracted from the three-dimensional point cloud data can be fully utilized. Then, according to the initial local grid features, the local grid features of each target and the global grid features, the aggregated features of the three-dimensional target are obtained, which fully utilizes the features extracted from different modal data, thereby improving the accuracy and robustness of three-dimensional target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a multimodal detection model provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a process flow of a first three-dimensional target detection method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a flow chart of a second three-dimensional target detection method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a process flow of a third three-dimensional target detection method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a fourth three-dimensional target detection method provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a fifth three-dimensional object detection method provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of a sixth three-dimensional object detection method provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of a seventh three-dimensional object detection method provided in an embodiment of the present application;

[0050] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0052] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0053] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0054] Optionally, the three-dimensional object detection method provided in the embodiment of the present application is applied to an electronic device, which may be a terminal device with computing and display capabilities, such as a mobile phone, a tablet computer, a laptop computer, a PDA, a desktop computer, or a server. Specifically, it may be applied to an application in a terminal device, such as an APP (application, mobile phone software) of a mobile phone, an application system on a computer, etc.

[0055] The specific implementation process of three-dimensional target detection provided in the embodiments of the present application is explained in detail below.

[0056] Figure 1 A schematic diagram of a multimodal detection model provided in an embodiment of the present application, wherein the multimodal detection model is installed in the aforementioned electronic device, such as Figure 1 As shown, the multimodal detection model may include a feature extraction module, a learnable function local fusion module, a point cloud image global fusion module, and a border and confidence prediction module. Each module in the multimodal detection model uses the method provided in the embodiment of the present application to detect three-dimensional objects, so that three-dimensional objects can be accurately identified.

[0057] Optionally, a multimodal detection model can be pre-trained in an electronic device. Specifically, sample data of different three-dimensional targets in a manufacturing scene can be collected as a sample data set. The three-dimensional targets can include different categories such as people, machines, and objects. The sample data of each three-dimensional target can include a two-dimensional image and three-dimensional point cloud data of each three-dimensional target.

[0058] Optionally, the sample data set is divided into a 60% training set and a 40% validation set, the training set is input into the multimodal detection model, the multimodal detection model is back-propagated and trained to update the parameters of the multimodal detection model, and after the optimized multimodal detection model is obtained, the validation set is input into the multimodal detection model for verification, the target detection result is obtained, and the evaluation is performed. Finally, the optimized multimodal detection model after evaluation is used as the pre-trained multimodal detection model.

[0059] Figure 2 The flowchart of the first three-dimensional target detection method provided in the embodiment of the present application is shown in FIG. 1 , wherein the execution subject of the method is the aforementioned electronic device. Figure 2 As shown, the method includes:

[0060] S101 , acquiring each two-dimensional image of a three-dimensional target and three-dimensional point cloud data corresponding to each two-dimensional image, and determining at least one non-empty voxel and a non-empty voxel feature corresponding to the non-empty voxel according to each three-dimensional point cloud data.

[0061] Optionally, the three-dimensional target can be different types of three-dimensional targets such as people, machines, objects, etc. The three-dimensional target can be photographed from different perspectives to obtain two-dimensional images of the three-dimensional target at different perspectives, and a set of three-dimensional point cloud data corresponding to the two-dimensional image can also be obtained using a three-dimensional scanning device.

[0062] Optionally, after obtaining the three-dimensional point cloud data of the two-dimensional image at different viewing angles, at least one non-empty voxel can be determined based on each group of three-dimensional point cloud data. Specifically, each group of three-dimensional point cloud data can be voxelized first, and the voxels with point cloud data can be used as non-empty voxels, and feature extraction can be performed on the point cloud data in each non-empty voxel to obtain the non-empty voxel features corresponding to each non-empty voxel. The non-empty voxels obtained from all groups of three-dimensional point cloud data and the set of all non-empty voxel features are used as the non-empty voxel feature set corresponding to the three-dimensional target.

[0063] Optionally, Among them, t ve is the non-empty voxel feature, N v is the number of non-empty voxels in each 3D proposal box, V e is the voxel index, T v It is the non-empty voxel feature set corresponding to each set of 3D point cloud data, that is, the non-empty voxel feature set corresponding to each 3D suggestion box described below.

[0064] S102, input each two-dimensional image and each three-dimensional point cloud data into a feature extraction module of a pre-trained multimodal detection model, and obtain at least one initial two-dimensional image feature corresponding to each two-dimensional image and a three-dimensional suggestion box corresponding to each three-dimensional point cloud data.

[0065] The three-dimensional suggestion box may be used to indicate a three-dimensional area where a three-dimensional object exists.

[0066] Optionally, the feature extraction module may include a two-dimensional feature extraction unit and a three-dimensional feature extraction unit. The two-dimensional image can be input into the two-dimensional feature extraction unit for feature extraction to obtain at least one initial two-dimensional image feature of the two-dimensional image. Then, multiple initial two-dimensional image features can be obtained for one two-dimensional image, and different initial two-dimensional image features can represent different features.

[0067] Optionally, the 3D point cloud data corresponding to the 2D image is input into the 3D feature extraction unit, and the 3D suggestion box corresponding to the group of 3D point cloud data can be obtained. Specifically, the group of 3D point cloud data can be voxelized first, and the 3D feature extraction unit is used to extract the voxelized 3D point cloud data and generate the 3D suggestion box corresponding to the group of 3D point cloud data. Then, each 2D image corresponds to multiple 2D image features and a 3D suggestion box.

[0068] S103, inputting each 3D suggestion box and each initial 2D image feature into a learnable function local fusion module of a pre-trained multimodal detection model, dividing each 3D suggestion box and determining the initial local grid feature of each 3D suggestion box according to the division result, and determining the target local grid feature of each 3D suggestion box based on each initial 2D image feature and each initial local grid feature.

[0069] Optionally, each 3D suggestion box can be divided, and local mesh features can be extracted from the divided 3D suggestion boxes, so that the mesh features T corresponding to each 3D suggestion box can be obtained. A .

[0070] Specifically, a 3D suggestion box and each initial 2D image feature input value of the 2D image corresponding to the 3D suggestion box can be respectively input into a learnable function local fusion module of a multimodal detection model that is pre-trained to obtain the initial local grid features of the 3D suggestion box.

[0071] Optionally, the target local grid features of each 3D suggestion frame can be determined based on each initial 2D image feature corresponding to each 2D image and the initial local grid features of the 3D suggestion frame corresponding to each 2D image. The initial local grid features of each 3D suggestion frame are obtained by learning and processing using a learnable function local fusion module after dividing each 3D suggestion frame respectively. The feature granularity extracted from the initial local grid features of each 3D suggestion frame is finer and more accurate. Then, the target local grid features of each 3D suggestion frame obtained based on the initial local grid features of each 3D suggestion frame and the initial 2D image features corresponding to each 2D image are combined with the features extracted from the 2D image and the features extracted from the 3D point cloud data, so that the target local grid features are more accurate.

[0072] S104, input each non-empty voxel corresponding to each 3D suggestion box, each non-empty voxel feature, and each target 2D image feature corresponding to each 2D image into a point cloud image global fusion module in a pre-trained multimodal detection model for global grid feature processing to obtain a global grid feature corresponding to each 3D suggestion box.

[0073] Among them, the target two-dimensional image features are obtained by aggregating the semantic features of the initial two-dimensional image features in the learnable function local fusion module. As mentioned above, each two-dimensional image can correspond to multiple initial two-dimensional image features, or it can correspond to a three-dimensional suggestion box of three-dimensional point cloud data. In the learnable function local fusion module, each three-dimensional suggestion box is divided into multiple voxel grids. Therefore, the center point of each voxel grid after the three-dimensional suggestion box corresponding to the two-dimensional image is divided can be projected into the two-dimensional image, and then the semantic features are aggregated on the initial two-dimensional image features corresponding to the two-dimensional image through learnable dynamic offset, so that the target two-dimensional image features corresponding to each two-dimensional image can be obtained. The combination of each target two-dimensional image corresponding to each two-dimensional image can be obtained using T t To express.

[0074] Optionally, in the point cloud image global fusion module, the point cloud data of each 3D suggestion box and the 2D image corresponding to each 3D suggestion box are globally fused, and compared with the local grid feature processing in the learnable function fusion module, the obtained feature granularity is larger.

[0075] S105 , obtaining aggregate features of the three-dimensional target according to each initial local grid feature, each target local grid feature, and each global grid feature, and detecting the three-dimensional target according to the aggregate features.

[0076] Optionally, in the aggregated features, not only the initial local grid features with smaller feature extraction granularity are taken into account, but also the local grid features of each target obtained based on the initial local grid features and the initial two-dimensional image features, and the global grid features with larger feature extraction granularity are also taken into account. Therefore, the aggregated features of the obtained three-dimensional target are more accurate and make full use of the features extracted from different modal data, thereby improving the accuracy of three-dimensional target detection.

[0077] In this embodiment, each three-dimensional suggestion box and each initial two-dimensional image feature are input into the learnable function local fusion module of the pre-trained multimodal detection model, each three-dimensional suggestion box is divided and the initial local grid feature of each three-dimensional suggestion box is determined according to the division result, so that the obtained initial local grid feature granularity can be finer and more accurate, and the subtle features of the three-dimensional target are extracted, and the target local grid feature of each three-dimensional suggestion box is determined based on each initial two-dimensional image feature and each initial local grid feature. Then, the target local grid feature of the three-dimensional suggestion box combines the features extracted from different modal data, so that the obtained target local grid feature is more accurate. At the same time, the point cloud image global fusion module is used to perform global grid feature processing on the point cloud data and the two-dimensional image to obtain the global grid feature corresponding to each three-dimensional suggestion box. Then, through the processing of the two fusion modules, the features extracted from the two-dimensional image and the features extracted from the three-dimensional point cloud data can be fully utilized, and then the aggregated features of the three-dimensional target are obtained according to each initial local grid feature, each target local grid feature and each global grid feature, and the features extracted from different modal data are fully utilized, so that the accuracy and robustness of the three-dimensional target detection can be improved.

[0078] Figure 3 A schematic diagram of the process of the second three-dimensional target detection method provided in the embodiment of the present application is as follows: Figure 3 As shown, in the above S103, each 3D suggestion box and each initial 2D image feature are input into a learnable function local fusion module of a pre-trained multimodal detection model, each 3D suggestion box is divided and the initial local grid feature of each 3D suggestion box is determined according to the division result, and the target local grid feature of each 3D suggestion box is determined based on each initial 2D image feature and each initial local grid feature, which may include:

[0079] S201 , dividing each 3D suggestion box into a plurality of voxel grids, extracting local grid features according to each voxel grid, and obtaining initial local grid features of each 3D suggestion box.

[0080] Optionally, each 3D proposal box can be divided into multiple voxel grids G i , where i is the index of the voxel grid. For each voxel grid, local grid feature extraction is performed to obtain the grid feature of each voxel grid The set of mesh features of each voxel mesh of all bounding boxes is used as the initial local mesh features of the three-dimensional object.

[0081] Optionally, for each voxel grid, a position information encoder may be used to encode the position information of each voxel grid.

[0082] S202 , obtaining the image-enhanced local grid features of each 3D suggestion frame according to each target 2D image feature of the 2D image corresponding to each 3D suggestion frame and the initial local grid features of each 3D suggestion frame.

[0083] Specifically, the target 2D image features of the 2D image corresponding to each 3D suggestion frame can be subjected to centroid fusion processing with the initial local grid features of each 3D suggestion frame to obtain the image-enhanced local grid features of each 3D suggestion frame, that is, the target 2D image features are fused on the basis of the initial local grid features of each 3D suggestion frame. The image-enhanced local grid features of each 3D suggestion frame can be obtained, for example, using T z To express.

[0084] Exemplarily, for the three-dimensional suggestion box 1, each target two-dimensional image feature of the two-dimensional image 1 corresponding to the three-dimensional suggestion box 1 can be centroidally fused with the initial local grid features of the three-dimensional suggestion box 1 to obtain the image-enhanced local grid features of the three-dimensional suggestion box 1.

[0085] S203 , fusing the enhanced local grid features of each image and the initial local grid features to obtain target local grid features of each three-dimensional suggestion box.

[0086] Specifically, the local grid features T after the image enhancement of each 3D suggestion box can be z And the initial local mesh features T of each 3D proposal box A Perform fusion processing to obtain the fusion grid features T of each 3D suggestion box j1 , and then the fusion grid features T of each 3D proposal box j1 A feedforward neural network is used to reduce the number of channels, thereby obtaining the target local grid features T of each 3D proposal box. j .

[0087] Exemplarily, the local mesh features of the three-dimensional suggestion box 1 after image enhancement and the initial local mesh features of the three-dimensional suggestion box 1 may be fused to obtain the fused mesh features of the three-dimensional suggestion box 1 .

[0088] Figure 4 A schematic diagram of a process flow of a third three-dimensional target detection method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, in the above S201, each 3D suggestion box is divided into a plurality of voxel grids, and local grid features are extracted according to each voxel grid to obtain initial local grid features of each 3D suggestion box, which may include:

[0089] S301 , dividing each three-dimensional suggestion box to obtain a plurality of voxel grids corresponding to each three-dimensional suggestion box.

[0090] Specifically, each 3D suggestion box can be divided into u×u×u to obtain multiple voxel grids for each 3D suggestion box. The center point of each voxel grid in a bounding box can be calculated using Z i To indicate that the center point of each 3D suggestion box is represented by C z To express.

[0091] S302 , obtaining grid features of each voxel grid corresponding to each three-dimensional suggestion box according to the variant neural network formula and each voxel grid.

[0092] Specifically, it can be obtained by the following formula (1).

[0093]

[0094] Where δ = Z i -C z The center point Z of each voxel grid is i Relative to the center point C of the 3D suggestion box where the voxel grid is located z The relative position of is the number of point clouds in each voxel grid, and θ is the offset constant. KAN is specifically calculated by the following formula (II).

[0095]

[0096]

[0097] spline(x)=∑ i c i B n (x) Formula (IV)

[0098] Among them, w in formula (III) b The weight value obtained by Xavier initialization, w s The initial value of is 1, spline(x) is the spline function, c i are the coefficients of the spline function optimized during training, B n is the B-spline basis function defined on the voxel grid, x is the input matrix, specifically The matrix composed of It is the function matrix corresponding to the L-2th KAN layer, which is specifically obtained by the following formula (5) and formula (6).

[0099] When K = 0,

[0100] Among them, B n,0 (x) is the 0th order B-spline basis function of the nth node, t nIt is the nth node in the node sequence. The node is defined by the spline function. The function of the node is to divide the range of the independent variable x into several intervals.

[0101] When K≠0,

[0102] Among them, B n,k (x) is the kth order B-spline basis function of the nth node, t n is the nth knot in the knot sequence and k is the order of the spline.

[0103] Optionally, for each 3D suggestion box, the set of grid features of all voxel grids in each 3D suggestion box can be obtained by the above formula (I) to formula (VI) as the initial local grid feature T of the 3D suggestion box. A .

[0104] S303: taking a set of grid features of all voxel grids corresponding to each 3D suggestion box as an initial local grid feature of each 3D suggestion box.

[0105] Optionally, the initial local mesh feature of each three-dimensional suggestion box is a set of mesh features of all voxel meshes corresponding to the three-dimensional suggestion box.

[0106] In this embodiment, when extracting the grid features of each voxel grid, the KAN function is used instead of the MLP function. KAN uses a learnable activation function to replace the fixed activation function. Each weight parameter in the KAN function can be replaced by a univariate function. These functions are usually parameterized in the form of spline functions, thereby providing extremely high flexibility, fundamentally eliminating the dependence on the linear weight matrix, avoiding the influence of long-distance input on the output, and ensuring the stability of the detection results.

[0107] Figure 5 A schematic diagram of a fourth three-dimensional target detection method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, in the above S202, according to each target two-dimensional image feature of the two-dimensional image corresponding to each three-dimensional suggestion box and the initial local grid feature of each three-dimensional suggestion box, the local grid feature after image enhancement of each three-dimensional suggestion box is obtained, which may include:

[0108] S401 , fusing each target two-dimensional image feature of the two-dimensional image corresponding to each three-dimensional suggestion box with the initial local grid feature of the three-dimensional suggestion box to obtain a fused local grid feature.

[0109] Specifically, the cross attention mechanism can be used to first combine the target 2D image features of the 2D image corresponding to the 3D proposal box with the initial local grid features T of the 3D proposal box. AThe fused local mesh features of each 3D suggestion box are obtained by fusion. Since the initial local mesh features of each 3D suggestion box are the collection of mesh features of all voxel meshes in each 3D suggestion box, the fused local features of each 3D suggestion box contain the mesh features of all fused voxel meshes of each 3D suggestion box.

[0110] S402: Generate first query features corresponding to each 3D suggestion box according to each fused local grid feature and each initial local grid feature with position coding information.

[0111] Specifically, the first query features corresponding to each 3D suggestion box are generated from the fused local grid features of each 3D suggestion box and the initial local grid features of each 3D suggestion box with position coding information.

[0112] Optionally, since the initial local grid feature of each 3D suggestion box is a collection of grid features of all voxel grids of each 3D suggestion box, and each voxel grid generates the position information of the voxel grid in the foregoing, the grid features of the voxel grid in the initial local grid feature of each 3D suggestion box can be queried through the position information of the voxel grid. Since the fused local feature contains the grid features of all the fused voxel grids, the grid features of each fused voxel grid in the fused local feature can also be queried through the position information of the voxel grid.

[0113] Optionally, the first query feature corresponding to each generated three-dimensional suggestion box can be obtained using Q i To represent, i is the index number of each voxel grid in each 3D proposal box.

[0114] S403 : Obtain first sampling features corresponding to each two-dimensional image according to each target two-dimensional image feature of the two-dimensional image corresponding to each three-dimensional suggestion box and the sampling points corresponding to each three-dimensional suggestion box.

[0115] By sampling the 3D point cloud data in each 3D suggestion box, sampling points of each 3D suggestion box can be obtained, and the obtained reference points are key points in the 3D suggestion box. The first sampling feature is the sampled 2D image feature of each target.

[0116] Optionally, since a three-dimensional suggestion box corresponds to a group of three-dimensional point cloud data, and the group of three-dimensional point cloud data corresponds to a two-dimensional image, each target two-dimensional image feature of the two-dimensional image can be sampled in combination with each sampling point in the three-dimensional suggestion box of the three-dimensional point cloud data corresponding to the two-dimensional image to obtain the sampled target two-dimensional image features corresponding to the two-dimensional image.

[0117] Optionally, after sampling each target two-dimensional image feature of each two-dimensional image, the sampled target two-dimensional image features corresponding to each two-dimensional image are obtained, and then the first sampling feature T corresponding to each two-dimensional image is obtained. t1 , that is, the first sampling feature corresponding to each two-dimensional image is the feature of each target two-dimensional image after sampling.

[0118] S404, obtaining the local grid features of each three-dimensional suggestion frame after image enhancement according to each first query feature, each first sampling feature, the reference point of the centroid of each voxel grid in each three-dimensional suggestion frame on the two-dimensional image corresponding to the three-dimensional suggestion frame, the total number of sampling points in each three-dimensional suggestion frame, the total number of self-attention heads, the learnable weights, the first sampling offset and the first attention weight.

[0119] Specifically, it can be obtained by the following formula (VII).

[0120]

[0121] Among them, T z is the local grid feature after image enhancement of each 3D proposal box, S i is the reference point of the centroid of the i-th voxel grid in each 3D suggestion box on the 2D image corresponding to each 3D suggestion box, is the first sampling feature corresponding to the reference point of the centroid of the i-th voxel grid in each 3D proposal box on the 2D image, W M The sum is W′ q The learnable weights, M is the total number of self-attention heads, K is the total number of sampling points of each 3D proposal box, Δs qk is the first sampling offset of the kth sampling point by the qth self-attention head, A qk is the first attention weight value of the kth sampling point in the qth self-attention head. The first sampling offset and the first attention weight value can be obtained by linear projection of the first query feature corresponding to each 3D proposal box.

[0122] Optionally, the image enhancement can be performed on the grid features of each three-dimensional suggestion box using formula (VII) to obtain the image-enhanced grid features corresponding to each three-dimensional suggestion box.

[0123] Figure 6 A schematic diagram of a fifth three-dimensional target detection method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, in the above S104, each non-empty voxel corresponding to each 3D suggestion box, each non-empty voxel feature, and each target 2D image feature corresponding to each 2D image are input into the point cloud image global fusion module in the pre-trained multimodal detection model for global grid feature processing to obtain the global grid feature corresponding to each 3D suggestion box, which may include:

[0124] S501 , determining the point centroid of each non-empty voxel and the target point of the point centroid in the target two-dimensional image according to the three-dimensional point cloud data in each non-empty voxel corresponding to each three-dimensional suggestion box and the camera projection matrix.

[0125] The target image is a two-dimensional image corresponding to each three-dimensional suggestion box. For example, if the two-dimensional image corresponding to the three-dimensional suggestion box 1 is the two-dimensional image 1, then the target two-dimensional image corresponding to the three-dimensional suggestion box 1 is the two-dimensional image 1, then the point centroid of each non-empty voxel in the three-dimensional suggestion box 1 and the target point of each point centroid in the two-dimensional image 1 can be determined; if the two-dimensional image corresponding to the three-dimensional suggestion box 2 is the two-dimensional image 2, then the target two-dimensional image corresponding to the three-dimensional suggestion box 2 is the two-dimensional image 2, then the point centroid of each non-empty voxel in the three-dimensional suggestion box 2 and the target point of each point centroid in the two-dimensional image 2 can be determined.

[0126] S502. Obtain the image-enhanced voxel features corresponding to each three-dimensional suggestion box according to each target point in each target two-dimensional image, each target two-dimensional image feature corresponding to each target two-dimensional image, each non-empty voxel feature corresponding to each three-dimensional suggestion box, a learnable weight, a second sampling offset, and a second attention weight value.

[0127] Specifically, the above formula (VII) can be used to perform image enhancement on each voxel feature corresponding to each three-dimensional suggestion box to obtain the image-enhanced voxel feature corresponding to the three-dimensional suggestion box.

[0128] S503 , obtaining global grid features corresponding to each 3D suggestion box according to the enhanced voxel features of each image and the non-empty voxel features corresponding to each 3D suggestion box.

[0129] Specifically, the image-enhanced voxel features corresponding to each 3D suggestion box are cascaded with each non-empty voxel feature corresponding to each 3D suggestion box to obtain the fused voxel feature corresponding to each 3D suggestion box. Among them, N is the number of voxels corresponding to each 3D suggestion box, and is the number of channels of the neural network. Then, a feedforward neural network is used to reduce the number of channels and perform pooling processing to obtain the global grid feature T corresponding to each 3D suggestion box. p .

[0130] Figure 7 A flowchart of a sixth three-dimensional target detection method provided in an embodiment of the present application is shown as follows: Figure 7 As shown, in the above S501, according to the 3D point cloud data in each non-empty voxel corresponding to each 3D suggestion box and the camera projection matrix, determining the point centroid of each non-empty voxel and the target point of the point centroid in the target 2D image may include:

[0131] S601 , taking the average value of the spatial coordinates of all three-dimensional point cloud data in each non-empty voxel as the point centroid of the non-empty voxel.

[0132] Specifically, it can be calculated by the following formula (8).

[0133]

[0134] Among them, C e is the point centroid of each non-empty voxel, d e is the spatial coordinate of each 3D point cloud data in each non-empty voxel, |D(V e )| is the total number of point cloud data in each non-empty voxel, V e The index number of each non-empty voxel.

[0135] S602 , obtaining the target point of the centroid of each point in the target two-dimensional image according to the centroid of each point and the camera projection matrix.

[0136] Specifically, it can be obtained by the following formula (IX).

[0137] S e =F*C e

[0138] Among them, F is the product of the camera internal parameter matrix and the external parameter matrix, S e is the target point in the target 2D image at the point centroid of each non-empty voxel, C e is the point centroid of each non-empty voxel.

[0139] Optionally, there may be one or more non-empty voxels in a three-dimensional suggestion box. Then, through the aforementioned steps S601 to S602, one or more point centroids in the three-dimensional suggestion box may be obtained, and correspondingly, one or more target points may be included in the target two-dimensional image.

[0140] Figure 8 A flow chart of a seventh three-dimensional target detection method provided in an embodiment of the present application is shown as follows: Figure 8 As shown, in the above S502, according to each target point in each target two-dimensional image, each target two-dimensional image feature corresponding to each target two-dimensional image, each non-empty voxel feature corresponding to each three-dimensional suggestion box, a learnable weight, a second sampling offset, and a second attention weight value, the image-enhanced voxel feature corresponding to each three-dimensional suggestion box is obtained, which may include:

[0141] S701 , weighting processing is performed on each target two-dimensional image feature of each target point preset area in each target two-dimensional image to obtain an aggregated image feature corresponding to each target two-dimensional image.

[0142] Optionally, for a target point in a target two-dimensional image, each target two-dimensional image feature of a preset area range of the target point can be obtained, and then each target two-dimensional image feature of all target points in the target two-dimensional image is weighted to obtain an aggregate image feature T corresponding to the target two-dimensional image. t2 .

[0143] S702. Determine the image-enhanced voxel features corresponding to each three-dimensional suggestion box based on the aggregated image features corresponding to each target two-dimensional image, each target point on each target two-dimensional image, the total number of centroids of the midpoints of each three-dimensional suggestion box, each non-empty voxel feature corresponding to each three-dimensional suggestion box, the second sampling offset, and the second attention weight value.

[0144] Specifically, the first query feature Q in the above formula (VII) is i Replaced with each non-empty voxel corresponding to each 3D suggestion box, the first sampling feature Replaced by the aggregated image features corresponding to each target 2D image, S i is replaced by each target point on each target two-dimensional image, K is replaced by the total number of centroids of the midpoints in each three-dimensional proposal box, the first sampling offset is replaced by the second sampling offset, and the first attention weight value is replaced by the second weight value.

[0145] Optionally, in S105, obtaining an aggregate feature of the three-dimensional target according to each initial local grid feature, each target local grid feature, and each global grid feature, and detecting the three-dimensional target according to the aggregate feature may include:

[0146] Optionally, each initial local mesh feature is an initial local mesh feature T of each three-dimensional suggestion box. A , each target local mesh feature is the target local mesh feature T corresponding to each 3D suggestion box j , each global grid feature is the global grid feature T corresponding to each 3D suggestion box p .

[0147] Optionally, the initial local mesh features T corresponding to all 3D proposal boxes A The set of initial local mesh features T′ of the three-dimensional object A ; Target local mesh features T corresponding to all 3D proposal boxes j The set of the target local grid features T′ of the three-dimensional object j ; Global grid features T corresponding to all bounding boxes p The set of is the global grid feature T′ of the three-dimensional object p .

[0148] The aggregation features of the three-dimensional target are obtained by the following formula (10).

[0149] TH =T′ A +T′ j +T′ p Formula (X)

[0150] Optionally, after obtaining the aggregate features of the three-dimensional target, the relationship between different voxel grid features can be established through the self-attention mechanism, and the query feature Q, key K and value V can be generated from the aggregate features. The attention weights are calculated and weighted summation is performed to obtain the aggregate features processed by the self-attention mechanism. The specific calculation formula is as shown in the following formula (XI).

[0151]

[0152] Among them, d k is the dimension of the key vector, K T is the transposed matrix of the key matrix K.

[0153] Then, the input aggregate features and the aggregate features processed by the self-attention mechanism are sequentially added, residually connected, normalized, and nonlinearly transformed and processed through a two-layer feedforward neural network to obtain the target aggregate features of the three-dimensional object. The target aggregate features finally obtained can complete the interaction between the non-empty grid point features and the encoder layer and residual connection block of the standard transformer network. The interactive features are used as the target aggregate features of the three-dimensional target, which can be used to optimize the three-dimensional suggestion box.

[0154] Optionally, the loss function in the multimodal detection model is as shown in the following formula (XII).

[0155] L=β1L RPN +β2L cts +β3L reg Formula (12)

[0156] Among them, L is the total loss, β1, β2, β3 are hyperparameters for balancing different losses, and L cts is the prediction box classification loss, L reg is the bounding box regression loss, L RPN is the RPN loss.

[0157] Optionally, the target aggregation features of the three-dimensional target are input into a bounding box and confidence prediction module for detection to obtain a detection result.

[0158] Fig. 9 This is a structural block diagram of an electronic device 800 provided in an embodiment of the present application. The electronic device can be, for example, a three-dimensional target detector as described in the above embodiment. Fig. 9 As shown, the electronic device may include: a processor 801 and a memory 802 .

[0159] Optionally, a bus 803 may also be included, wherein the memory 802 is used to store machine-readable instructions executable by the processor 801. When the electronic device 800 is running, the processor 801 communicates with the memory 802 through the bus 803. When the machine-readable instructions are executed by the processor 801, the method steps in the above method embodiment are performed.

[0160] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method steps in the above-mentioned three-dimensional object detection method embodiment are executed.

[0161] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0162] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0163] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional target detection method, characterized in that: The method comprises: Acquire each two-dimensional image of the three-dimensional target and three-dimensional point cloud data corresponding to each of the two-dimensional images, and determine at least one non-empty voxel and a non-empty voxel feature corresponding to the non-empty voxel according to each of the three-dimensional point cloud data; Input each of the two-dimensional images and each of the three-dimensional point cloud data into a feature extraction module of a pre-trained multimodal detection model, respectively, to obtain at least one initial two-dimensional image feature corresponding to each of the two-dimensional images and a three-dimensional suggestion box corresponding to each of the three-dimensional point cloud data, wherein the three-dimensional suggestion box is used to indicate a three-dimensional area where the three-dimensional object exists; Inputting each of the three-dimensional suggestion boxes and each of the initial two-dimensional image features into the learnable function local fusion module of the pre-trained multimodal detection model, dividing each of the three-dimensional suggestion boxes and determining the initial local grid features of each of the three-dimensional suggestion boxes according to the division results, and determining the target local grid features of each of the three-dimensional suggestion boxes based on each of the initial two-dimensional image features and each of the initial local grid features; Inputting each non-empty voxel corresponding to each of the three-dimensional suggestion boxes, each non-empty voxel feature, and each target two-dimensional image feature corresponding to each two-dimensional image into the point cloud image global fusion module of the pre-trained multimodal detection model to perform global grid feature processing, so as to obtain the global grid feature corresponding to each of the three-dimensional suggestion boxes, wherein the target two-dimensional image feature is obtained by performing semantic feature aggregation on the initial two-dimensional image feature; According to each of the initial local grid features, each of the target local grid features and each of the global grid features, an aggregate feature of the three-dimensional target is obtained, and the three-dimensional target is detected according to the aggregate feature.

2. The three-dimensional target detection method according to claim 1, characterized in that: The dividing each of the three-dimensional suggestion boxes and determining the initial local grid features of each of the three-dimensional suggestion boxes according to the division results, and determining the target local grid features of each of the three-dimensional suggestion boxes based on each of the initial two-dimensional image features and each of the initial local grid features, comprises: Dividing each of the three-dimensional suggestion boxes into a plurality of voxel grids, extracting local grid features according to each of the voxel grids, and obtaining the initial local grid features of each of the three-dimensional suggestion boxes; Obtaining, according to the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes and the initial local grid features of each of the three-dimensional suggestion boxes, the image-enhanced local grid features corresponding to each of the three-dimensional suggestion boxes; The enhanced local grid features of each image and the initial local grid features are fused to obtain target local grid features of each three-dimensional suggestion box.

3. The three-dimensional target detection method according to claim 2, characterized in that: The step of dividing each of the three-dimensional suggestion boxes into a plurality of voxel grids and extracting local grid features according to each voxel grid to obtain the initial local grid features includes: Dividing each of the three-dimensional suggestion boxes to obtain a plurality of voxel grids corresponding to each of the three-dimensional suggestion boxes; According to the variant neural network formula and each of the voxel grids, grid features of each of the voxel grids corresponding to each of the three-dimensional suggestion boxes are obtained; A set of grid features of all voxel grids corresponding to each of the three-dimensional suggestion boxes is used as the initial local grid feature of each of the three-dimensional suggestion boxes.

4. The three-dimensional target detection method according to claim 2, characterized in that: The obtaining, according to the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes and the initial local grid features of each of the three-dimensional suggestion boxes, the image-enhanced local grid features of each of the three-dimensional suggestion boxes comprises: fusing the target two-dimensional image features of the two-dimensional image corresponding to each of the three-dimensional suggestion boxes with the initial local grid features of each of the three-dimensional suggestion boxes to obtain fused local grid features of each of the three-dimensional suggestion boxes; Generating first query features corresponding to each of the three-dimensional suggestion boxes according to each of the fused local grid features and each of the initial local grid features with position coding information; Obtaining first sampling features corresponding to each of the two-dimensional images according to each of the target two-dimensional image features of the two-dimensional images corresponding to each of the three-dimensional suggestion boxes and sampling points corresponding to each of the three-dimensional suggestion boxes; The image-enhanced local grid features of each of the three-dimensional suggestion boxes are obtained according to each of the first query features, each of the first sampling features, a reference point of the centroid of each voxel grid in each of the three-dimensional suggestion boxes on the two-dimensional image corresponding to the three-dimensional suggestion box, the total number of sampling points in each of the three-dimensional suggestion boxes, the total number of self-attention heads, the learnable weights, the first sampling offset and the first attention weight value.

5. The three-dimensional target detection method according to claim 1, characterized in that: The step of inputting each non-empty voxel, each non-empty voxel feature and each target two-dimensional image feature corresponding to each of the three-dimensional suggestion boxes into the point cloud image global fusion module of the multimodal detection model for global grid feature processing to obtain the global grid feature corresponding to each of the three-dimensional suggestion boxes includes: Determine, according to the 3D point cloud data in each non-empty voxel corresponding to each of the 3D suggestion boxes and the camera projection matrix, the point centroid of each of the non-empty voxels and the target point of the point centroid in a target 2D image, wherein the target 2D image is a 2D image corresponding to each of the 3D suggestion boxes; Obtaining, according to each target point in each target two-dimensional image, each target two-dimensional image feature corresponding to each target two-dimensional image, each non-empty voxel feature corresponding to each three-dimensional suggestion box, a learnable weight, a second sampling offset, and a second attention weight value, an image-enhanced voxel feature corresponding to each three-dimensional suggestion box; According to the voxel features after each image enhancement and the non-empty voxel features corresponding to each three-dimensional suggestion box, the global grid features corresponding to each three-dimensional suggestion box are obtained.

6. The three-dimensional target detection method according to claim 5, characterized in that: The determining, according to the three-dimensional point cloud data in each non-empty voxel corresponding to the three-dimensional suggestion box and the camera projection matrix, the point centroid of each non-empty voxel and the target point of the point centroid in the target two-dimensional image comprises: Taking the average value of the spatial coordinates of all three-dimensional point cloud data in each of the non-empty voxels as the point centroid of the non-empty voxel; According to the centroids of the points and the camera projection matrix, the target points of the centroids of the points in the target two-dimensional image are obtained.

7. The three-dimensional target detection method according to claim 5, characterized in that: The method of obtaining the image-enhanced voxel features corresponding to each of the three-dimensional suggestion boxes according to each of the target points in each of the target two-dimensional images, each of the target two-dimensional image features corresponding to the target two-dimensional images, each of the non-empty voxel features corresponding to each of the three-dimensional suggestion boxes, a learnable weight, a second sampling offset value, and a second attention weight value, comprises: Performing weighted processing on each target two-dimensional image feature of each target point preset area in each target two-dimensional image to obtain an aggregated image feature corresponding to each target two-dimensional image; According to the aggregated image features corresponding to each of the target two-dimensional images, each target point on each of the target two-dimensional images, the total number of point centroids in each of the three-dimensional suggestion boxes, each non-empty voxel feature corresponding to each of the three-dimensional suggestion boxes, the second sampling offset and the second attention weight value, the image-enhanced voxel features corresponding to each of the three-dimensional suggestion boxes are determined.

8. The three-dimensional target detection method according to claim 1, characterized in that: The step of obtaining the aggregated features of the three-dimensional target according to the initial local grid features, the target local grid features and the global grid features includes: The initial local grid features, the target local grid features and the global grid features are weightedly added to obtain the aggregated features.

9. An electronic device, characterized in that: It comprises a memory and a processor, the memory stores a computer program executable by the processor, and the processor implements the steps of the three-dimensional target detection method described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional object detection method according to any one of claims 1 to 8 are executed.

Images