A cross-modal 3D target detection method based on hypergraph correlation

Abstract

The application provides a cross-modal 3D target detection method based on hypergraph correlation, comprising obtaining point cloud data L and image data C, and further comprising the following steps: selecting a key region; constructing a local-global feature hypergraph; propagating the local-global hypergraph; aligning multi-scale high-quality features; and regressing an inter-scale hypergraph.The application provides a cross-modal 3D target detection method based on hypergraph correlation, which is used for extracting multi-granularity semantic information of a scene, exploring the representation of high-order semantic content between key regions, constructing a hypergraph for inter-scale cross-modal key regions, and directly regressing the related attributes of objects in a three-dimensional space based on the implicit representation of high-order semantic content of the objects by the key regions, so that the accuracy of three-dimensional object detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D target detection, and in particular to a cross-modal 3D target detection method based on hypergraph association. BACKGROUND

[0002] 3D target detection can identify and locate objects in the surrounding environment, and such spatial information is beneficial to improving the decision-making ability of an autonomous driving system. Meanwhile, an unmanned vehicle is equipped with multiple sensors such as radars and cameras, and the fusion of cross-modal information can effectively improve the perception ability of 3D target detection to the environment.

[0003] Some current cross-modal fusion methods use semantic labels of image pixels to decorate corresponding point cloud data, or generate dense virtual point clouds based on the nearest association of image pixels and point clouds, and use image information to enhance the expression of point cloud data to scene information. Meanwhile, a shared BEV space can be used to represent point cloud and image features at the same time, and feature-level fusion can be achieved. The above methods realize cross-modal fusion through projection or splicing methods, but image information is relatively rich and dense, and point cloud information is relatively sparse. The heterogeneity of such modal data will cause a large number of aligned features to have content misalignment, missing and inconsistency. In addition, some other methods use attention mechanisms to model the feature fusion process to alleviate the alignment deviation. However, not all features need to be focused on, and this way of calculating the correlation between all cross-modal features will increase the computational cost. This makes some current methods still limited to the level of feature fusion, lacking high-level semantic understanding of the content of the scene environment.

[0004] Chinese patent application with publication number CN114663880A discloses a three-dimensional target detection method based on a multi-level cross-modal self-attention mechanism, which includes constructing a training set and a test set using RGB image data; constructing a three-dimensional target detection model, which includes an RGB backbone network, a depth backbone network, a classifier, and a regressor; training the three-dimensional target detection model using the training set and the test set data, and verifying the training effect using the test set to obtain a trained three-dimensional target detection model; and detecting three-dimensional targets in the RGB image using the trained model. The disadvantage of this method is the lack of learning and representation of high-level semantic content between multi-scale key regions, which affects the performance of target detection. SUMMARY

[0005] In order to solve the above technical problems, the application provides a cross-modal 3D target detection method based on hypergraph association, which is used for extracting multi-granularity semantic information of a scene, exploring representation of high-order semantic content between key regions, constructing a hypergraph for cross-modal key regions between scales, and directly regressing related attributes of objects in a three-dimensional space through implicit representation of high-order semantic content of the objects based on the key regions, so as to improve the accuracy of three-dimensional object detection.

[0006] The application provides a cross-modal 3D target detection method based on hypergraph association, which comprises acquiring point cloud data L and image data C, and further comprises the following steps:

[0007] Step 1: selecting key regions;

[0008] Step 2: constructing a local-global feature hypergraph;

[0009] Step 3: propagating the local-global hypergraph;

[0010] Step 4: aligning multi-scale high-quality features;

[0011] Step 5: regressing the hypergraph between scales.

[0012] Preferably, the step 1 comprises the following sub-steps:

[0013] Step 11: for the point cloud data L and the image data C={C1, C2, …, C N , point cloud feature extraction is performed through a radar encoder to obtain point cloud BEV features , and image feature extraction is performed using an image encoder to obtain image BEV features B C and B L are subjected to information aggregation to obtain basic cross-modal fusion features B base , and the formula is

[0014] B base =Concat(B C ,B L );

[0015] Step 12: for the cross-modal fusion features B base , a key region selection function f select is used to dynamically select important regions in a cross-modal scene represented by B base to obtain K cross-modal key regions r={r1, r2, …, r K}, and r1, r2, …, r K =f select (B base );

[0016] Step 13: define the key region selection function as L2 norm;

[0017] where N is the number of camera sensors used to collect image information around the vehicle, W and H are the number of BEV grids in the x-axis and y-axis, is the dimension of the feature channel, K is the number of cross-modal key regions.

[0018] In any of the above schemes, preferably, the step 13 includes converting to where n is WxH, i.e. the number of BEV grids, calculate the L2 norm of the feature vector for each BEV grid, and take the top K grid regions with the largest L2 norm as the key regions, and the corresponding features as the key region features B K ,

[0019] r1,r2,…,r K = topk(||B′ base,1 ||2,||B′ base,2 ||2,…,||B′ base,n ||2)

[0020] where topk is the index position of the top K largest elements.

[0021] In any of the above schemes, preferably, the step 2 includes dividing the BEV feature B base into P patches, and the BEV feature of the pth patch is convert to where n p is

[0022] In any of the above schemes, preferably, the step 2 further includes performing cross-modal key region selection on the p patches, taking the global key region r g as the vertex set of the hypergraph , taking the feature corresponding to the global key region as the global vertex feature B g,K , constructing the hyperedge set ε g of the global hypergraph, and defining the correlation matrix H g of ,

[0023]

[0024] where B O,p is the overall feature representing the pth patch, and i is the number of grids in a patch.

[0025] Preferably, in any of the above schemes, the method for selecting the cross-modal critical region includes the following sub-steps:

[0026] Step 21: Construct a local hypergraph at the local level For the p-th patch, perform cross-modal key region selection, and select the local key region r of the p-th patch. l p As a hypergraph vertex set Features of the corresponding local key areas Vertex features of a local hypergraph;

[0027] Step 22: Using a feature-based approach, search for the T nearest neighbor vertices of each vertex using KNN to construct a hyperedge set.

[0028]

[0029] in, For the vertex The T nearest vertices;

[0030] Step 23: Implement a local hypergraph The construction and definition of the hypergraph Correlation matrix as follows:

[0031]

[0032] in, Let i be the i-th vertex in the p-th local hypergraph. Let j be the hyperedge in the p-th local hypergraph.

[0033] In any of the above schemes, step 3 preferably includes based on and The vertex features of the p-th local hypergraph As input, a hypergraph neural network is used for local hypergraph propagation to obtain features with local high-order correlations.

[0034]

[0035] Where σ is the activation function, and Let be the diagonal matrices representing the vertex degree and hyperedge degree of the p-th local hypergraph, respectively. and Let be the hyperedge weights and learnable vertex weights in the p-th local hypergraph.

[0036] In any of the above schemes, step 3 preferably includes a method based on... With H g The vertex features B of the global hypergraph g,K The feature G is obtained by global hypergraph propagation as input, resulting in a feature G with global high-order correlations. g ={G g,1 ,…,G g,p ,…,G g,P};

[0037] Will With G g By combining these features, we obtain higher-order correlation features that contain both local and global information. and G lg Restore to the original BEV feature form.

[0038]

[0039] In any of the above schemes, step 4 preferably includes according to B. base Extracting multi-scale features Multiscale features Perform intra-scale high-order association modeling to obtain multi-scale high-order association features with multi-granular semantic information of the scene.

[0040] In any of the above schemes, step 4 preferably includes... Calculate heatmaps to obtain high-quality alignment features at different scales With the corresponding cross-modal key region get

[0041]

[0042] In any of the above schemes, step 5 preferably includes constructing an inter-scale hypergraph. Will As a hypergraph vertex set The corresponding vertex features are And construct the hyperedge set ε it .

[0043] In any of the above schemes, step 5 preferably further includes using the inter-scale hypergraph. Implicit representation of high-order semantics of objects in multi-scale, cross-modal key regions, based on hypergraph structure. Perform hypergraph regression to directly output the relevant attributes of the target object.

[0044]

[0045] where W F are learnable regression weights, D it,v and D it,e are diagonal matrices of inter-scale hypergraph vertex and hyperedge degrees, W it,e and θ it are hyperedge weights and learnable vertex weights in inter-scale hypergraph , and H it is the adjacency matrix of .

[0046] The present application proposes a cross-modal 3D object detection method based on hypergraph association, which uses a hypergraph propagation method to extract the inherent complex semantic relationship in cross-modal data, and improves the performance of 3D object detection. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a preferred embodiment of the cross-modal 3D object detection method based on hypergraph association according to the present application.

[0048] Figure 2 is a flowchart of another preferred embodiment of the cross-modal 3D object detection method based on hypergraph association according to the present application.

[0049] Figure 3 is a schematic diagram of an embodiment of the key region selection process of the cross-modal 3D object detection method based on hypergraph association according to the present application.

[0050] Figure 4 is a schematic diagram of an embodiment of the local-global hypergraph construction process of the cross-modal 3D object detection method based on hypergraph association according to the present application.

[0051] Figure 5 is a schematic diagram of an embodiment of the local-global hypergraph propagation process of the cross-modal 3D object detection method based on hypergraph association according to the present application.

[0052] Figure 6 is a schematic diagram of an embodiment of the inter-scale hypergraph regression process of the cross-modal 3D object detection method based on hypergraph association according to the present application. DETAILED DESCRIPTION

[0053] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0054] Embodiment One

[0055] The application provides a cross-modal 3D target detection method based on hypergraph association, which is used for extracting multi-granularity semantic information of a scene and exploring representation of high-order semantic content between key regions. First, an intra-scale hypergraph propagation method is based on multi-scale cross-modal scene regions of a cross-modal BEV grid representation, and dynamically selects key regions. Intra-scale regions are divided into multiple blocks, and learning of multi-scale block high-order association is realized through construction and propagation of a local-global hypergraph, so that multi-granularity high-order semantic information of the scene is represented. Secondly, an inter-scale hypergraph regression method is based on a heat map to obtain a specific number of multi-scale high-quality alignment features, so as to reduce the influence of a large number of alignment deviation features on modeling. A hypergraph is constructed for inter-scale cross-modal key regions, and by using implicit representation of high-order semantic content of objects based on the key regions, relevant attributes of the objects in a three-dimensional space are directly regressed, so that the accuracy of three-dimensional object detection can be improved.

[0056] As shown in Figure 1 , a cross-modal 3D target detection method based on hypergraph association comprises the following steps.

[0057] Step 110 is performed to select key regions, including the following sub-steps:

[0058] In step 111, for the point cloud data L and the image data C = {C1, C2, …, C N}, point cloud feature extraction is performed through a radar encoder to obtain point cloud BEV features , and image feature extraction is performed using an image encoder to obtain image BEV features . Information aggregation is performed on B C and B L to obtain basic cross-modal fusion features B base , and the formula is

[0059] B base = Concat(B C , B L ).

[0060] In step 112, for the cross-modal fusion features B base , a key region selection function f select is used to dynamically select important regions in a cross-modal scene represented by B base , to obtain K cross-modal key regions r = {r1, r2, …, r K}, and r1, r2, …, r K = f select (B base ).

[0061] In step 113, the key region selection function is defined as an L2 norm, and converted into wherein n is WxH, i.e. the number of BEV grids, the L2 norm of the feature vector is calculated for each BEV grid, and the top K grid regions with the largest L2 norm are taken as the key regions, and the corresponding features are taken as the key region features B K ,

[0062] r1, r2,..., r K = topk(||B′ base，1 ||2, ||B′ base，2 ||2,..., ||B′ base,n ||2)

[0063] wherein N is the N camera sensors used to collect image information around the ego vehicle, W and H are the number of BEV grids in the x-axis and y-axis, is the feature channel dimension, K is the number of cross-modal key regions, and topk is the index position of the top K largest elements.

[0064] Step 120 is performed to construct a local-global feature supergraph, and the BEV features B base are divided into P patches, and the BEV features of the pth patch are converted into wherein n p is

[0065] Cross-modal key region selection is performed on the P patches, and the global key region r g is taken as the vertex set of the supergraph The features corresponding to the global key region are taken as the global vertex features B g,K , the hyperedge set ε g of the global supergraph is constructed, and the correlation matrix H g of the global supergraph is defined as ,

[0066]

[0067] wherein B O,p is the overall feature representing the pth patch, and i is the number of grids in a patch.

[0068] The cross-modal key region selection method includes the following sub-steps:

[0069] Step 121 is performed to construct a local supergraph from the local level, and cross-modal key region selection is performed on the pth patch to obtain the local key region of the pth patch​ As a hypergraph vertex set Features of the corresponding local key areas Vertex features of a local hypergraph;

[0070] Step 122 involves using a feature-based approach, employing KNN to search for the T nearest neighbor vertices to each vertex, and constructing a hyperedge set.

[0071]

[0072] in, For the vertex The T nearest vertices;

[0073] Perform steps 123 to achieve a local hypergraph. The construction and definition of the hypergraph Correlation matrix as follows:

[0074]

[0075] in, Let i be the i-th vertex in the p-th local hypergraph. Let j be the j-th hyperedge in the p-th local hypergraph.

[0076] Execute step 130 to propagate the local-global hypergraph, based on and The vertex features of the p-th local hypergraph As input, a hypergraph neural network is used for local hypergraph propagation to obtain features with local high-order correlations.

[0077]

[0078] Where σ is the activation function, and Let be the diagonal matrices representing the vertex degree and hyperedge degree of the p-th local hypergraph, respectively. and Let be the hyperedge weights and learnable vertex weights in the p-th local hypergraph.

[0079] based on With H g The vertex features B of the global hypergraph g,K The features are used as input for global hypergraph propagation to obtain features with global high-order associations.

[0080] Will With G gBy combining these, we obtain a high-order correlation feature G that contains both local and global information. lg ={G lg,1 ,…,G lg,p ,…,G lg,P}, and G lg Restore to the original BEV feature form.

[0081]

[0082] Perform step 140 to align multi-scale high-quality features, based on B base Extracting multi-scale features Multiscale features Perform intra-scale high-order association modeling to obtain multi-scale high-order association features with multi-granular semantic information of the scene.

[0083] right Calculate heatmaps to obtain high-quality alignment features at different scales With the corresponding cross-modal key region get

[0084]

[0085] Perform step 150 to regress the inter-scale hypergraph and construct the inter-scale hypergraph. Will As a hypergraph vertex set The corresponding vertex features are And construct the hyperedge set ε it .

[0086] Based on interscale hypergraphs Implicit representation of high-order semantics of objects in multi-scale, cross-modal key regions, based on hypergraph structure. Perform hypergraph regression to directly output the relevant attributes of the target object.

[0087]

[0088] Among them, W F For learnable regression weights, D it,v and D it,e Hypergraphs between scales W is the diagonal matrix of mid-vertex degree and hyperedge degree. it,e and θ it Hypergraphs between scales The hyperedge weights and learnable vertex weights in H it for The correlation matrix.

[0089] The present application aims to propose a new cross-modal 3D target detection network, which uses a hypergraph propagation method to extract the complex semantic relationship inherent in cross-modal data, and improves the performance of 3D target detection.

[0090] Embodiment two

[0091] The present application discloses a scale-in hypergraph propagation and scale-inter hypergraph regression method, which is used to extract multi-granularity semantic information of a scene and explore the representation of high-order semantic content between key regions. First, the scale-in hypergraph propagation method is based on multi-scale cross-modal scene regions of cross-modal BEV grid representation, and dynamically selects key regions. The scale-in region is divided into multiple blocks, and the learning of multi-scale block high-order correlation is realized through the construction and propagation of local-global hypergraph, representing the multi-granularity high-order semantic information of the scene. Second, the scale-inter hypergraph regression method obtains a certain number of multi-scale high-quality alignment features based on the heat map, and reduces the influence of a large number of alignment deviation features on modeling. The hypergraph is constructed for the scale-inter cross-modal key region, and the implicit representation of the object high-order semantic content based on the key region is directly regressed to the related attributes of the object in three-dimensional space, which can improve the accuracy of three-dimensional object detection.

[0092] As shown in Figure 2 , the technical solution is as follows:

[0093] 1. Key region selection

[0094] For the input point cloud data L and image data C={C1,C2,…,C N}, where N is the number of N camera sensors used to collect image information around the vehicle. The point cloud feature is extracted by the radar encoder to obtain the point cloud BEV feature and the image feature is extracted by the image encoder to obtain the image BEV feature where W and H are the number of BEV grid in the x-axis and y-axis, is the feature channel dimension. The information aggregation of B C and B L obtains the basic cross-modal fusion feature B base .

[0095] B base =Concat(B C ,B L ) (1)

[0096] For the cross-modal fusion feature B base , the key region selection function f select is used to dynamically select the important regions in the cross-modal scene represented by B base , and K cross-modal key regions r={r1,r2,…,rK}, excluding the influence of content dislocation, missing and inconsistency existing in other cross-modal regions on the model performance.

[0097] r1,r2,…,r K =f select (B base ) (1)

[0098] The key region selection function is defined as the L2 norm. In this step, the is converted into where n is WxH, i.e. the number of BEV grids. The L2 norm of the feature vector is calculated for each BEV grid, and the top K grid regions with the largest L2 norm are selected as the key regions, and the corresponding features are taken as the key region features B K .

[0099] r1,r2,…,r K =topk(||B′ base,1 ||2,||B′ bese,2 ||2,...,||B′ base,n ||2)#(3)

[0100] where topk represents the position index of the top K largest elements.

[0101] 2. Local-global hypergraph construction

[0102] The BEV feature B base is divided into P patches, and the BEV feature of the pth patch is The is converted into where n p is For the pth patch, a local hypergraph H is first constructed at the local level The local key region of the pth patch obtained by cross-modal key region selection is taken as the vertex set V of the hypergraph H The feature corresponding to the local key region is taken as the vertex feature of the local hypergraph. In a feature-based manner, the T nearest neighbor vertices of each vertex are searched by KNN, and the hyperedge set E

[0103]

[0104] where V is the T nearest neighbor vertices of the vertex V . Thus, the local hypergraph H is constructed The construction and definition of the hypergraph Correlation matrix as follows:

[0105]

[0106] in, Let i be the i-th vertex in the p-th local hypergraph. Let j be the j-th hyperedge in the p-th local hypergraph. Next, construct the global hypergraph at the global level. By aggregating the features of each BEV grid in each patch, we obtain features that represent the overall information of each patch.

[0107] Similarly, the above method is used to select cross-modal key regions for p patches, and the global key region r is selected. g As a hypergraph vertex set Features corresponding to the global key regions are used as global vertex features B g,K Construct the hyperedge set ε of the global hypergraph according to Formula 4. g and define The correlation matrix is ​​H g .

[0108]

[0109] 3. Local-to-Global Hypergraph Propagation

[0110] based on and The vertex features of the p-th local hypergraph As input, a hypergraph neural network is used for local hypergraph propagation to obtain features with local high-order correlations.

[0111]

[0112] Where σ is the activation function, and Let be the diagonal matrices representing the vertex degree and hyperedge degree of the p-th local hypergraph, respectively. and Let be the hyperedge weights and learnable vertex weights in the p-th local hypergraph.

[0113] Similarly, based on and The vertex features B of the global hypergraph g,K The feature G is obtained by global hypergraph propagation as input, resulting in a feature G with global high-order correlations. g ={G g,1 ,…,Gg,p ,…,G g,P}.Will With G g By combining these, we obtain a high-order correlation feature G that contains both local and global information. lg ={G lg,1 ,…,G lg,p ,…,G lg,P}, and G lg Restore to the original BEV feature form.

[0114]

[0115] 4. High-quality feature alignment at multiple scales

[0116] Because features at a single scale often lack sufficient granularity in information representation, according to B... base Extracting multi-scale features Based on the aforementioned local-global hypergraph construction and propagation methods, multi-scale features... Perform intra-scale high-order association modeling to obtain multi-scale high-order association features with multi-granular semantic information of the scene.

[0117] Due to the differences between cross-modal data, many alignment features exhibit alignment bias. Directly using the fused features obtained in this step for object detection will negatively impact model performance. Therefore, [further adjustments are needed]. Calculate heatmaps to obtain high-quality alignment features at different scales With the corresponding cross-modal key region

[0118]

[0119] 5. Hypergraph Regression Between Scales

[0120] Cross-modal key regions at different scales represent scene content and objects differently. For example, pedestrians, bicycles, and cars have different scales, but understanding the semantic content of the scene composed of these three types of objects helps improve the model's detection performance. Therefore, further constructing inter-scale hypergraphs is crucial. Will As a hypergraph vertex set The corresponding vertex features are And construct the hyperedge set ε according to Formula 4. it .

[0121] Based on interscale hypergraphs Implicit representation of high-order semantics of objects in multi-scale, cross-modal key regions, based on hypergraph structure. Perform hypergraph regression to directly output the relevant attributes of the target object.

[0122]

[0123] where W F are learnable regression weights. D it,v and D it,e are diagonal matrices of vertex degrees and hyperedge degrees in inter-scale hypergraph , respectively. W it,e and θ it are hyperedge weights and learnable vertex weights in inter-scale hypergraph . H it is the adjacency matrix of .

[0124] The beneficial effects of the present application include:

[0125] (1) The method constructs intra-scale local and global hypergraphs according to the dynamically selected cross-modal key regions, and models the local and global high-order correlations between them, thereby helping to learn the complex semantic relationships inherent in cross-modal data.

[0126] (2) The method obtains multi-scale high-quality alignment features based on heat maps, constructs inter-scale hypergraphs for the corresponding multi-scale key regions, and directly regresses objects to extract high-order semantic information of inter-scale features, which can realize high-performance fusion regression.

[0127] Embodiment Three

[0128] As shown in Figure 3 , for the input BEV feature, each grid in it can be regarded as a region in the original scene, so the BEV feature can be converted by feature conversion, and is converted into where n is WxH, that is, the number of BEV grids, and thus the BEV region can be obtained. By taking the L2 norm as the key region selection function, the L2 norm of the feature vector corresponding to each BEV grid in the BEV region is calculated, and the grid with the largest norm of the first K feature vectors can be regarded as the key region.

[0129] Embodiment Four

[0130] As shown in Figure 4As shown, the BEV region is divided into P patches, each patch representing a local region in the scene. At the local level, local key regions are selected for each patch, and these local key regions serve as vertices of the local hypergraph. The BEV mesh features corresponding to these local key regions are used as vertex features. This process is used to construct P local hypergraphs. At the global level, the BEV mesh features in each patch are summed to achieve feature aggregation. Treating a patch as a region, the aggregated features of that region are used as vertex features, and global key regions are selected to obtain a global hypergraph.

[0131] Example 5

[0132] like Figure 5 As shown, hypergraph convolution is applied to the constructed P local hypergraphs and one global hypergraph to propagate the local and global hypergraphs, resulting in local and global features with high-order correlations. The vertex features from the global hypergraph are then added to each vertex of the corresponding local hypergraph, thereby achieving feature fusion between the local and global hypergraphs and obtaining fused features with high-order local-global correlations.

[0133] Example 6

[0134] like Figure 6 As shown, based on the BEV features obtained from feature extraction, a local-global hypergraph construction and propagation method is used to obtain multi-scale local-global high-order features. The importance of alignment features at different scales is calculated using a heatmap, and the L2 norm is used to select the top K most important features at different scales as high-quality multi-scale alignment features. High-quality regions and high-quality alignment features at different scales are used as vertices and vertex features of the inter-scale hypergraph, and the inter-scale hypergraph is constructed and propagated to obtain features with high-order inter-scale correlations. Based on the inter-scale hypergraph structure, the obtained features are directly used to predict the relevant attributes of the target object, thereby achieving hypergraph regression and obtaining the 3D bounding boxes of the relevant objects.

[0135] To better understand this invention, specific embodiments have been described in detail above, but these are not intended to limit the invention. Any simple modifications made to the above embodiments based on the technical essence of this invention still fall within the scope of this invention. Each embodiment in this specification focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to mutually. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A cross-modal 3D object detection method based on hypergraph association, comprising obtaining point cloud data L and image data C, characterized in that, Further comprising the following steps: Step 1: Selecting key regions; Step 2: Construct a local-global feature hypermap, integrating the BEV features B base Divided into P patches, the BEV feature of the p-th patch is: Will Convert to Where, n p for Cross-modal key region selection is performed on p patches, and the global key region r is selected. g As a hypergraph vertex set Features corresponding to the global key regions are used as global vertex features B g,K Construct the hyperedge set ε of the global hypergraph g and define The correlation matrix is ​​H g , where B O,p is the overall feature of the pth patch, and i is the number of meshes in a patch. The method for selecting the cross-modal key regions comprises the following sub-steps: Step 21: Construct a local hypergraph at the local level For the p-th patch, perform cross-modal key region selection, and select the local key region r of the p-th patch. l p As a hypergraph vertex set Features of the corresponding local key areas Vertex features of a local hypergraph; Step 22: In a feature-based way, the KNN is used to search the T nearest neighbors of each vertex, and the hyperedge set is constructed wherein, are the T vertices closest to the vertex in distance. Step 23: Realize the local hypergraph of the construction, and define the incidence matrix of the hypergraph as follows:​ wherein, is the i-th vertex in the p-th local hypergraph, is the j-th hyperedge in the p-th local hypergraph; Step 3: Propagate local global hypergraph, based on With The vertex features of the p-th local hypergraph As input, employ hypergraph neural network for local hypergraph propagation to obtain features with local high-order correlation where σ is an activation function, and are the diagonal matrices of vertex degree and hyperedge degree in the p-th local hypergraph, respectively, and are the hyperedge weight in the p-th local hypergraph and the learnable vertex weight, respectively. based on With H g The vertex features B of the global hypergraph g,K The feature G is obtained by global hypergraph propagation as input, resulting in a feature G with global high-order correlations. g ={G g,1 ,…,G g,p ,…,G g,P };Will With G g By combining these, we obtain a high-order correlation feature G that contains both local and global information. lg ={G lg,1 ,…,G lg,p ,…,G lg,P }, and G lg Restore to the original BEV feature form. Step 4: Aligning multi-scale high-quality features; Step 5: Regressing inter-scale hypergraphs.

2. The hypergraph association based cross-modal 3D object detection method of claim 1, wherein, The step 1 comprises the following sub-steps: Step 11: For the point cloud data L and the image data C = {C1, C2, …, Cn}, point cloud feature extraction is performed by a radar encoder to obtain point cloud BEV features N and image feature extraction is performed using an image encoder to obtain image BEV features B C and B L are aggregated to obtain the basic cross-modal fusion feature B base , and the formula is​ B base = Concat(B C ,B L ) Step 12: For the cross-modal fusion feature B base , the important regions in the cross-modal scene represented by B select are dynamically selected using a key region selection function f base , to obtain K cross-modal key regions r = {r1, r2, …, r K}, r1, r2, …, r K = f select (B base ); Step 13: Defining the key region selection function as an L2 norm; Wherein, N is the N camera sensors used to collect the image information around the vehicle, W and H are the number of BEV grid on the x-axis and y-axis, is the feature channel dimension, K is the number of cross-modal key regions.

3. The hypergraph association based cross-modal 3D object detection method of claim 2, wherein, The step 13 includes converting into where n is WxH, i.e., the number of BEV grids, the L2 norm of the feature vector is calculated for each BEV grid, and the K largest grid regions with the L2 norm are taken as key regions, and the corresponding features are taken as key region features B K , r1, r2,..., r K = topk(||B'1||2, ||B'2||2,..., ||B'k||2) base,1 = topk(||B'1||2, ||B'2||2,..., ||B'k||2) base,2 = topk(||B'1||2, ||B'2||2,..., ||B'k||2) base , n = topk(||B'1||2, ||B'2||2,..., ||B'k||2 Wherein, topk is the index position of the top K largest elements.

4. The hypergraph association based cross-modal 3D object detection method of claim 3, wherein, The step 4 comprises obtaining the multi-scale feature according to B base extracting multi-scale features extracting multi-scale features performing intra-scale high-order correlation modeling to obtain multi-scale feature high-order correlation features with scene multi-granularity semantic information 5. The hypergraph association based cross-modal 3D object detection method of claim 4, wherein, The step 4 further comprises obtaining the high-quality aligned features at different scales by computing a heat map computing a heat map to obtain high-quality aligned features at different scales corresponding cross-modal key regions obtaining

Images