Small object feature extraction method based on Visual-Transform network

Abstract

The invention belongs to the technical field of deep learning, and discloses a Visual-Transform network-based small object feature extraction method, which comprises the steps of 1, performing global feature extraction based on a self-attention mechanism, fusing self-attention feature learning into 3D feature extraction network construction, and effectively combining the self-attention feature learning with a convolutional neural network to obtain a 3D feature extraction model; more detailed target information is reserved in the shallow network, and peripheral features outside the center of the target are filtered out; 2, based on a key point sampling model of prior information, dividing an original point cloud space into four equal parts, carrying out parallel calculation on the position of each point in a candidate region, directly discarding points outside the candidate region, and only calculating the characteristics of key points in the candidate region; and step 3, fusing the global features and the local features, extracting the local features by using the global features to provide prior information, and then integrating the local features into a global feature map of a three-dimensional space. According to the invention, self-attention three-dimensional object representation is realized.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a small object feature extraction method based on a Visual-Transformer network. Background Art

[0002] With the rapid development of 3D acquisition technology, various types of 3D scanners, LiDAR, and RGB-D cameras are becoming increasingly accessible and cost-effective. These sensors provide 3D data rich in geometric details, shape features, and scale information. Combined with 2D images, 3D data provides machines with the opportunity to better understand their surroundings. 3D data has a wide range of applications, including but not limited to autonomous driving, robotics, remote sensing, and medicine. Specifically, in the field of autonomous driving, 3D data has become a key research area, laying the foundation for progress in this field. Therefore, how to effectively process 3D data and extract relevant features to facilitate tasks such as obstacle avoidance in autonomous driving has become a pressing issue facing researchers in this field.

[0003] 3D object detection algorithms must meet detection requirements in all weather conditions. Complex lighting conditions and dim night scenes create relatively poor data quality and blurred target texture information, placing higher demands on object detection algorithms. Therefore, there is a pressing need to focus on strategies that balance detection performance while also improving the ability to extract robust target features from point clouds. Research on 3D object detection in autonomous driving scenarios requires not only improving the accuracy of detection algorithms to adapt to various climates and environmental conditions, but also overcoming challenges in feature fusion and network structure design from the perspective of deep learning detection algorithms. Summary of the Invention

[0004] The purpose of the present invention is to provide a small object feature extraction method based on a Visual-Transformer network to solve the above technical problems.

[0005] To solve the above technical problems, the specific technical solution of the small object feature extraction method based on the Visual-Transformer network of the present invention is as follows:

[0006] A small object feature extraction method based on a Visual-Transformer network includes the following steps:

[0007] Step 1: Global feature extraction based on self-attention mechanism: Utilize the attention mechanism in the Visual Transformer mechanism for feature extraction and learn features by correlating features of adjacent pixels. Integrate self-attention feature learning into the construction of a 3D feature extraction network and effectively combine it with a convolutional neural network to retain more detailed target information in the shallow network and filter out peripheral features outside the target center.

[0008] Step 2: Key point sampling based on prior information: By dividing the original point cloud space into four equal parts, the position of each point in the candidate area is calculated in parallel, and the points outside the candidate area are directly discarded. Only the features of the key points in the candidate area are calculated;

[0009] Step 3: Fusion of global features and local features. The fusion method consists of two stages: the first stage provides prior information for extracting local features using global features, and the second stage integrates local features into the global feature map in three-dimensional space.

[0010] Furthermore, the step 1 includes:

[0011] The point cloud data is divided into volume voxels in a regular 3D space to accommodate the input of the global feature extraction network. The point cloud information is encoded and serialized through regularization processing.

[0012] Based on the voxelization method, the voxel block size is established in the three-dimensional coordinate system, and the size is (r x , r y , r z ), for different data sets, different voxel block sizes are set according to the three-dimensional space distance of the scene to reduce the loss of feature points during processing;

[0013] After determining the number of voxel blocks, each voxel block uses random sampling to collect 5 points of information to represent the voxel. If the number of point clouds is insufficient, repeated sampling is performed. At the same time, empty voxels are replaced with fixed values. In this way, after voxelization, a regularized three-dimensional point cloud feature matrix is ​​obtained.

[0014] Furthermore, the step 1 includes:

[0015] The 3D Transformer downsampling module adopts a sparse convolutional neural network as the key structure of the feature downsampling stage, uses sparse matrices and compressed representations to store and process sparse weights, and uses sparse matrix-vector multiplication and specialized convolution operations to accelerate calculations, thereby completing feature extraction and fusion.

[0016] Furthermore, the step 1 includes:

[0017] The overall process of sparse convolutional neural network is:

[0018] (1) Input data: Input voxelized point cloud data;

[0019] (2) Sparse voxel coding: Sparse voxel hashing is used as the coding method for processing sparse voxels. The coding process is as follows:

[0020] Store the position and feature values ​​of non-empty voxels in a hash table, where each hash table entry consists of a position key and a feature key.

[0021]

[0022] Where p is the three-dimensional position vector of the voxel, w is the random hash vector, s is the random offset, N is the size of the hash table, and H(p) represents the encoding result;

[0023] Query the value h(p) of a specific voxel position to determine the position of the feature point in the voxel.

[0024] h(p)=H(p) (8)

[0025] The output of the code is V hashcode This involves retrieving the locations and feature values ​​of all non-empty voxels in the hash table, and performing no operations on empty voxels.

[0026] V hashcode =(p,f)∈H(p), p≠null (9)

[0027] (3) Perform feature extraction using convolution operations: convolution is optimized for situations where there are only a few non-zero voxels;

[0028] (4) Decoding the encoded voxels: After feature extraction, the encoded voxels need to be decoded to restore the original point cloud representation. Formula (3) is used to complete the feature decoding operation to obtain the final feature map. Further, step 1 includes:

[0029] The feature extraction network introduces a self-attention mechanism in the submanifold voxel module, which is implemented through parallel computing. In the attention mechanism, local attention and global attention are used simultaneously to learn the feature relations around the target area.

[0030] The feature extraction network takes the feature map as input and consists of three network downsampling modules. Each sampling module contains a layer of convolutional neural network and two layers of sub-manifold voxel modules. The sub-manifold voxel module is the core structure of the feature extraction network. It converts point cloud data into voxel representation, which is a discrete three-dimensional voxel representation method. It also maps point cloud data to a low-dimensional space to reduce the dimensionality of point cloud data.

[0031] The structure of the submanifold voxel module is self-attention processing and submanifold voxel convolution. It adopts a multi-head attention mechanism, including voxel attention and global attention. It distributes the calculation of related areas to different attention layers. Submanifold voxel convolution is used to process voxelized point clouds.

[0032] Furthermore, the calculation process of the self-attention mechanism is as follows:

[0033] (1) Linear transformation of input data: For the input voxelized point cloud, three different linear transformation matrices are used, and the weight values ​​are (W Q , W K , W V ), map it to Query, Key, and Value to represent different target objects in the attention mechanism;

[0034] (2) Linear data segmentation: The query, key, and value representations obtained from the input data are split and assigned to voxel attention and global attention. Each attention layer processes a part of the input sequence, allowing the model to learn different attention representations in parallel to capture the feature relationship between voxel and global features respectively;

[0035] (3) Weight calculation: For each attention layer, the weight value is calculated using formula (4) to obtain the attention weight between each position in the voxel and other positions; then, by weighting the input sequence with the attention weight, the voxel blocks that are more closely related to the current position in the input sequence are highlighted with higher weights.

[0036]

[0037] Among them F attention is the output feature of self-attention, N non-empty The number of non-empty input voxelized point clouds. c , key embedding K c Sum value embedding V c It helps to extract the features C around the voxel center of the object region, and d is the voxel distance between two points;

[0038] (4) Output results: The output information of the two attention layers is concatenated and mapped back to the original voxelized point cloud matrix to obtain the feature results calculated by the attention mechanism.

[0039] Furthermore, the sub-manifold voxel convolution calculation process is as follows:

[0040] (1) Non-empty voxel selection: traverse the non-empty voxels in the voxelized point cloud data, and for each non-empty voxel, locate its position in the input data;

[0041] (2) Convolutional feature sampling: Convolutional feature extraction is applied around the input data. The size and shape of the convolution kernel are consistent with the convolution kernel used in traditional convolution operations. In submanifold convolution, the convolution kernel is only calculated on non-empty voxels, and each element in the convolution kernel needs to be multiplied by its corresponding non-empty voxel.

[0042] (3) Feature contribution calculation: After multiplying the elements in the convolution kernel with the corresponding non-empty voxels, the results are added to obtain the feature output value. This output value represents the new feature of the current non-empty voxel after the sub-manifold convolution operation. At the same time, the target center is set according to the annotation, and the feature contribution scores of adjacent non-empty voxels are calculated to filter out features far away from the target center and improve the representation of local detail features of the target; (4) Iterative calculation loop: Finally, iterate through all data, repeat the feature calculation, and then determine the final global target feature output.

[0043] Furthermore, the sub-manifold voxel convolution calculation process is as follows:

[0044] The three-dimensional Transformer downsampling network architecture consists of three network blocks, each of which contains a sparse convolutional neural network layer and two sub-manifold voxel modules. The sparse convolutional neural network implements the function of downsampling the target. In the process of global feature extraction, downsampling calculations are performed at three scales. At the same time, sub-manifold voxel modules are constructed for feature extraction at each sampling scale.

[0045] The network layer data within the sub-manifold voxel module is connected across layers to maintain feature weight sharing. The sub-manifold voxel module is used to redesign the non-empty voxel processing method. In this layer, the central non-empty voxel of each input data is determined, and the relevant non-empty voxel association relationship is calculated based on the set central voxel. The voxel feature attention module calculates the impact of non-empty voxels on target detection decisions and obtains the confidence score of each voxel. The global feature attention module divides the entire image into n regions and calculates the correlation between each region and the global feature map. In addition, this layer only calculates the non-empty voxels in each region. Through the self-attention layer, the feature weight score of each non-empty voxel is obtained, and the confidence score of the non-empty voxel is calculated twice in the single-layer sub-manifold network.

[0046] Furthermore, the step 2 includes:

[0047] Key point sampling uses a method of subdividing 3D point clouds and uses parallel computing to implement sampling processing. The original point cloud input data is spatially divided into 4 subsets, and key point sampling is performed on each subset separately. Parallel computing is used to accelerate key point sampling.

[0048] Use the candidate box generated in the first stage as the supervision area, only sample the key points of the point cloud within the area, and do not calculate the points outside the area.

[0049] The overall process of key point sampling consists of two stages. The first stage is to divide the point cloud space, and the second stage is to sample key points based on the prior box. The process is as follows:

[0050] (1) Space division: Quarter space: Let S be the original point cloud space, and use K to represent the number of subdivided subspaces. The original point cloud space is divided into K equal parts, with K = 4, and the division is performed on the plane. Each subspace independently and in parallel calculates the key points in the area. The formula for space division is written as:

[0051]

[0052] Among them, S K is the result of spatial partitioning, K is the number of subdivision subsets, (arctan(py i , px i )∈(-π,π) is the angle of rotation along the positive direction of the X axis, (py i , px i ) is the final rotation angle, p i is the output of the point set obtained from the parallel computation;

[0053] (2) Candidate point sampling: Based on the detection method, a candidate region is generated and key point sampling is performed in a limited area. Let P be the candidate region and P′ be the key point information retained within the candidate region after screening. The calculation formula is as follows:

[0054]

[0055] where p i From formula (5), the point cloud is represented, and the center area and scale size of each region are recorded as C and D respectively, c j is the center point of region C, (dx j ,dy j , dz j ) represents the length, width and height of the region, r (s) is the minimum radius of the area center point extending outward, P′ is the set of key points selected in the candidate area after point extraction,

[0056] For the selection of key points in the region, GPU is used to accelerate the calculation. At the same time, according to the calculation process of step (1), the 3D space is divided into 4 subsets for the initial selection of key points.

[0057] Furthermore, the step 3 includes:

[0058] In the process of key point feature extraction, the multi-layer perceptron uses three blocks to complete the feature downsampling process. Each block contains two layers, namely the feature map fusion layer and the multi-layer perceptron.

[0059] The feature map fusion layer includes the K nearest neighbor layer and the maximum pooling layer.

[0060] The multi-scale feature map in the three-dimensional Transformer downsampling network architecture is used as the supervision information for local key point sampling, and K nearest neighbors are used for feature selection to retain more effective information to supervise the key point sampling process. In terms of feature supervision, K is set to the number of channels of the multi-layer perceptron input data. For the input feature matrix, the Euclidean distance of all eigenvalues ​​is calculated. According to the set K value, the K nearest eigenvalues ​​are selected from the distance calculation results as representative features. Finally, we combine the selected K values ​​with the input feature point cloud as the basis for feature sampling. x represents the point cloud input after candidate sampling, and p1 represents the feature of the voxel convolution in this layer. Figure, the KNN method will complete the alignment and sampling of the input (x, p1), and perform normalization after maximum pooling to ensure that the specified number of points are input. The multilayer perceptron consists of two linear calculation layers and a fully connected layer. For this layer, the output of the previous layer (y, p2) is used for linear fusion calculation to meet the input requirements of the fully connected layer. Finally, this layer outputs the result (y1, p3) after network sampling. In this method, the output features are called key points, which are used to supplement the features of the second detection prediction; the global feature and local feature fusion method uses the RoI pooling of the region of interest to fuse global and local features and extract the target region of interest from the point cloud data.

[0061] The overall calculation process of RoI pooling is as follows:

[0062] (1) Based on the feature mapping matrix obtained by convolution of the original voxels, the voxel units corresponding to the candidate region are marked as the region of interest, the candidate region is adjusted, the position and size of the region of interest are determined, and only the marked region of interest is subjected to feature fusion. Therefore, only the voxel units within the region of interest are marked as valid regions, while the remaining voxel units are marked as invalid regions;

[0063] (2) Divide the voxel units in the effective area into a fixed number of sub-regions. RoI pooling divides the voxel units in the effective area into a fixed number of sub-regions;

[0064] (3) Perform a pooling operation on the voxel units in each sub-region. For each sub-region, RoI pooling performs a pooling operation on the feature vectors of all voxel units in the sub-region. In this work, the maximum pooling method is used to fuse local point cloud features with global features, and finally obtain the feature vector of each sub-region;

[0065] (4) According to the region of interest, the feature vectors of each sub-region are connected. The RoI pool connects the feature vectors of each sub-region to obtain the feature vector of the entire region of interest. This feature vector can be used as the input for subsequent target classification and regression tasks.

[0066] The small object feature extraction method based on the Visual-Transformer network of the present invention has the following advantages:

[0067] When used, the present invention uses a voxelization method and a three-dimensional Transformer module to realize global scene feature extraction, and adopts an efficient key point sampling method to sample local key point features. Finally, the RoI pooling model is used to realize the fusion of these two features. The three-dimensional Transformer module is designed to extract features from non-empty voxels, thereby reducing redundant calculations to a certain extent and making the target feature modeling process more closely aligned with the intrinsic information of the target. At the same time, in the global feature extraction process, sparse voxel hashing is used as the voxel encoding method, and the probability of hash conflicts is minimized by selecting a suitable hash function, thereby improving the storage efficiency and query speed of the model.

[0068] In the local feature extraction phase, the model's speed is further improved by using point cloud spatial segmentation to enable parallel computation of keypoint sampling. Targeted keypoint sampling is performed based on previously proposed candidate regions, creating an attention mechanism for the detailed areas of the target. The proposed attention mechanism and feature fusion method recombines the feature extraction method for objects in images, effectively addressing the problem of shallow neural networks losing information about small-scale object features. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flowchart of the feature extraction network based on Visual-Transformer of the present invention;

[0070] Figure 2 Schematic diagram of sparse convolutional neural network processing of the present invention;

[0071] Figure 3 Schematic diagram of the sub-manifold voxel module of the present invention;

[0072] Figure 4 Schematic diagram of the main body of the 3D Transformer downsampling module of the present invention;

[0073] Figure 5 This is a schematic diagram of the key point sampling module flow of the present invention;

[0074] Figure 6 This is a schematic diagram of the key point sampling process of the present invention;

[0075] Figure 7 The network and structure diagram of the MLP feature extraction of the present invention;

[0076] Figure 8 Schematic diagram of the process of the region of interest (RoI) pooling layer of the present invention. DETAILED DESCRIPTION

[0077] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a small object feature extraction method based on a Visual-Transformer network of the present invention in conjunction with the accompanying drawings.

[0078] See also Figure 1-8 , this application provides a small object feature extraction method based on Visual-Transformer network, including the following steps:

[0079] This method uses the attention mechanism in the Visual Transformer to extract global features. It learns features by correlating features from adjacent pixels to enhance the robustness of global features. Self-attention feature learning is integrated into the construction of a 3D feature extraction network and effectively combined with a convolutional neural network. This method retains more detailed target information in shallow layers and filters out peripheral features outside the target center, enabling the network to more effectively learn the overall feature information of the scene.

[0080] Keypoint sampling based on prior information: This method divides the original point cloud space into four equal parts, parallelizes the calculation of the position of each point within the candidate region, directly discards points outside the candidate region, and only calculates the features of the keypoints within the candidate region. This processing method utilizes parallel computing acceleration and prior information parameter adjustment methods to effectively improve the sampling efficiency of keypoints while also preserving local target characteristics, which is more conducive to the implementation of real-time feature extraction methods.

[0081] The fusion method involves two stages: the first stage provides prior information for extracting local features using global features, and the second stage integrates the local features into the global feature map in three-dimensional space. Throughout the entire process, a feature supervision mechanism is established to associate global and local features during the sampling process, thereby enhancing the robustness and generalization of the feature extraction network.

[0082] In order to achieve global feature extraction and point cloud information encoding, in this embodiment, preferably, the point cloud data is divided into volume voxels in a regular 3D space to accommodate the input of the global feature extraction network. Through regularization processing, the point cloud information can be effectively encoded and serialized;

[0083] Based on the voxelization method, the voxel block size is established in the three-dimensional coordinate system, and the size is (r x , r y , r z ), for different data sets, it is necessary to set different voxel block sizes according to the three-dimensional space distance of the scene to reduce the loss of feature points during the processing process.

[0084] After determining the number of voxel blocks, each block is randomly sampled to collect five points of information to represent that voxel. If the number of points in the cloud is insufficient, repeated sampling is performed. At the same time, empty voxels are replaced with fixed values. This results in a regularized 3D point cloud feature matrix after voxelization.

[0085] In order to achieve efficient feature extraction and fusion, in this embodiment, preferably, the three-dimensional Transformer downsampling module adopts a sparse convolutional neural network as the key structure of the feature downsampling stage, uses sparse matrices and compressed representations to store and process sparse weights, and uses sparse matrix-vector multiplication and specialized convolution operations to accelerate calculations, thereby completing feature extraction and fusion.

[0086] According to statistics of non-empty voxels, the non-empty voxel area usually only accounts for 10-20% of the global voxels. Therefore, most voxel blocks do not contain point cloud feature information, resulting in strong overall sparsity. Sparse convolutional neural networks are needed to quickly process sparse point cloud data, which achieves better performance in global feature representation and target feature downsampling. The overall process of sparse convolutional neural networks is as follows:

[0087] (1) Input data: Input voxelized point cloud data.

[0088] (2) Sparse voxel encoding: Sparse voxel hashing is used as an encoding method for processing sparse voxels. The principle is to store the positions and feature values ​​of non-empty voxels in a hash table in memory to reduce storage and calculation. The encoding process is as follows:

[0089] The position values ​​and feature values ​​of non-empty voxels are stored in a hash table, where each hash table entry consists of a position key and a feature key.

[0090]

[0091] Where p is the three-dimensional position vector of the voxel, w is the random hash vector, s is the random offset, N is the size of the hash table, and H(p) represents the encoding result.

[0092] By querying the value h(p) at a specific voxel position, the position of the feature point in the voxel can be effectively determined.

[0093] h(p)=H(p) (14)

[0094] The output of the code is V hashcode This involves retrieving the locations and feature values ​​of all non-empty voxels in the hash table, while performing no operations on empty voxels.

[0095] V hashcode =(p,f)∈H(p),p≠null (15)

[0096] (3) Use convolution operations to perform feature extraction: Convolution can be optimized for cases where there are only a few non-zero voxels, avoiding calculations for all voxels. This greatly reduces the amount of computation required and improves training and inference speed. By using convolution, more effective features can be extracted while maintaining data sparsity, thereby enhancing model performance. Since the data has been sparsely processed, convolution used for feature extraction can also be called sparse convolution.

[0097] (4) Decoding the encoded voxels: After feature extraction, the encoded voxels need to be decoded to restore the original point cloud representation. Formula (3) is used to complete the feature decoding operation and obtain the final feature map. Therefore, the entire processing process can be summarized as feature encoding, feature processing, feature decoding, feature fusion, and finally the output of the feature map.

[0098] To achieve more accurate point cloud feature extraction and enhancement, in this embodiment, the feature extraction network preferably introduces a self-attention mechanism in the submanifold voxel module, which is implemented through parallel computing. In this attention mechanism, local attention and global attention are used simultaneously to learn the feature relationships around the target area, achieving better feature information extraction.

[0099] The feature extraction network takes the feature map as input and consists of three network downsampling modules. Each downsampling module includes a convolutional neural network layer and two submanifold voxel modules. The submanifold voxel module is the core structure of the feature extraction network. It converts point cloud data into a voxel representation, a discrete 3D voxel representation method, and maps the point cloud data into a low-dimensional space, reducing its dimensionality.

[0100] The main structure of the submanifold voxel module is self-attention processing and submanifold voxel convolution. First, the self-attention mechanism helps neural networks better capture different features and relationships in the input sequence. It uses a multi-head attention mechanism, consisting of voxel attention and global attention, to distribute computations for relevant regions to different attention layers. Submanifold voxel convolution is used to process voxelized point clouds. Unlike traditional convolution operations, submanifold convolution only operates on non-empty voxels, improving computational efficiency and reducing resource consumption.

[0101] The calculation process of the self-attention mechanism is as follows:

[0102] (1) Linear transformation of input data: For the input voxelized point cloud, three different linear transformation matrices (weight values ​​are (W Q , W K , W V )), map it to Query, Key, and Value to represent different target objects in the attention mechanism.

[0103] (2) Linear Data Splitting: The query, key, and value representations obtained from the input data are split and assigned to voxel attention and global attention. Each attention layer processes a portion of the input sequence, allowing the model to learn different attention representations in parallel to capture the feature relationships between voxels and global features respectively.

[0104] (3) Weight calculation. For each attention layer, the weight value is calculated using formula (4) to obtain the attention weight between each position in the voxel and other positions. Then, by weighting the input sequence with the attention weight, the voxel blocks that are more closely related to the current position in the input sequence can be highlighted with higher weights.

[0105]

[0106] Among them F attention is the output feature of self-attention, N non-empty The number of non-empty input voxelized point clouds. c , key embedding K c Sum value embedding V c It helps to extract the features C around the voxel center of the object region, and d is the voxel distance between two points.

[0107] (4) Output results. The output information of the two attention layers is concatenated and mapped back to the original voxelized point cloud matrix to obtain the feature results calculated by the attention mechanism.

[0108] The submanifold voxel convolution calculation process is as follows:

[0109] (1) Non-empty voxel selection: traverse the non-empty voxels in the voxelized point cloud data. For each non-empty voxel, locate its position in the input data (spatial information).

[0110] (2) Convolutional feature sampling: Convolutional feature extraction is applied around the input data, and the size and shape of the convolution kernel are consistent with the convolution kernel used in traditional convolution operations (3×3 kernel). Therefore, in submanifold convolution, the convolution kernel is only calculated on non-empty voxels, and each element in the convolution kernel needs to be multiplied by its corresponding non-empty voxel.

[0111] (3) Feature contribution calculation. After multiplying the elements in the convolution kernel with their corresponding non-empty voxels, the results are added to obtain the feature output value. This output value represents the new feature of the current non-empty voxel after the submanifold convolution operation. At the same time, the target center is set according to the annotation, and the feature contribution scores of adjacent non-empty voxels are calculated to filter out features far away from the target center, thereby improving the representation of local detail features of the target.

[0112] (4) Iterative calculation loop. Finally, iterate through all data, repeat feature calculation, and then determine the final global target feature output.

[0113] To achieve more efficient point cloud feature reduction and extraction, in this embodiment, the preferred 3D Transformer downsampling network architecture consists of three network blocks, each of which includes a sparse convolutional neural network layer and two sub-manifold voxel layers. The sparse convolutional neural network implements the function of downsampling the target. During the global feature extraction process, downsampling calculations are performed at three scales. Simultaneously, a sub-manifold voxel module is constructed for feature extraction at each sampling scale to improve feature sampling efficiency and quality.

[0114] By cross-layer connection of network layer data within the sub-manifold voxel module and sharing of feature weights, the amount of computational parameters can be better reduced while improving the reusability of features.

[0115] Because the voxelized input data contains a large number of empty voxels, repeated feature sampling and calculation for these voxels is unnecessary. Therefore, to reduce computational complexity, a redesigned non-empty voxel processing method was implemented using the submanifold voxel module. The submanifold voxel module is a typical transformation-based method for processing non-empty voxel features. In this layer, the central non-empty voxel of each input data point is determined, and the association relationship between the relevant non-empty voxels is calculated based on the set central voxel. The voxel feature attention module calculates the influence of non-empty voxels on object detection decisions, generating a confidence score for each voxel. The global feature attention module divides the entire image into n regions and calculates the correlation between each region and the global feature map. Furthermore, this layer only considers the non-empty voxels in each region. Through the self-attention layer, a feature weight score for each non-empty voxel is obtained, which is helpful for the subsequent construction of the global feature model. To further refine the accuracy of subsequent feature fusion, the confidence score for non-empty voxels is calculated twice within the single-layer submanifold network.

[0116] To achieve efficient point cloud key point extraction and accelerated processing, in this embodiment, the key point sampling module preferably utilizes a method for subdividing 3D point clouds and uses parallel computing to implement sampling processing. This method spatially divides the original point cloud input data into four subsets and performs key point sampling on each subset separately, accelerating key point sampling through parallel computing.

[0117] The features extracted by sparse convolution have a weak ability to represent local information of the point cloud, which can reduce the accuracy of target positioning. Therefore, sampling key points from the original point cloud to represent local features and fusing global and local features are the key to solving the problem of information loss in sparse convolution features.

[0118] In order to improve the efficiency of key point sampling, the candidate box generated in the first stage is used as the supervision area, and key points are sampled only on the point cloud within the area, without calculating points outside the area.

[0119] The overall process of key point sampling consists of two stages. The first stage is to divide the point cloud space, and the second stage is to sample key points based on the prior box. The process is as follows:

[0120] (1) Space partitioning (quarter space): Let S be the original point cloud space, and use K to represent the number of subdivided subspaces. Divide the original point cloud space into K equal parts (using K = 4) and perform the division on the plane. Each subspace independently and in parallel calculates the key points in the region. The formula for space partitioning can be written as:

[0121]

[0122] Among them, S K is the result of spatial partitioning, K is the number of subdivision subsets, (arctan(py i , px i )∈(-π,π] is the angle of rotation along the positive direction of the X axis, (py i , px i ) is the final rotation angle, p i The output of the point set obtained from parallel computing. This processing method can achieve parallel point cloud sampling calculation, which speeds up the key point sampling process to a certain extent. In addition, since each subset point cloud obtained is independent, each subset point cloud can be run in parallel on the GPU, further speeding up the key point sampling process of the point cloud.

[0123] (2) Candidate point sampling: Based on the detection method, candidate regions are generated and key point sampling is performed within a limited area. This method is more supervised for local feature extraction and is more conducive to extracting detailed information. Let P be the candidate region and P′ be the key point information retained within the candidate region after screening. The calculation formula is as follows:

[0124]

[0125] where p i From formula (5), it represents the point cloud. The center area and scale size of each region are denoted as C and D respectively, c jis the center point of region C, (dx j ,dy j , dz j ) represents the length, width and height of the region, r (s) is the minimum radius of the region center point extending outward. The reason for taking the minimum value is that the search radius cannot exceed the boundary of the region. P′ is the set of key points selected in the candidate region after point extraction.

[0126] For the selection of key points within the region, a GPU can be used to accelerate the calculation. At the same time, according to the calculation process of step (1), the 3D space is divided into 4 subsets for the initial selection of key points. This process eliminates a large number of unnecessary feature points, which is more conducive to the subsequent key point feature extraction.

[0127] In order to achieve accurate feature dimensionality reduction and key point feature extraction, in this embodiment, preferably, the multilayer perceptron uses three blocks to complete the feature downsampling process during the key point feature extraction process, and each block contains two layers, namely the feature map fusion layer and the multilayer perceptron.

[0128] The feature map fusion layer includes the K nearest neighbor layer and the maximum pooling layer.

[0129] The multi-scale feature maps in the 3D Transformer downsampling network architecture are used as supervision information for local keypoint sampling. Due to the relatively large parameters of its feature maps, feature selection is required. In addition, the dimensions of the selected features do not match the input required by the MLP network. Therefore, K-nearest neighbors are used for feature selection to retain more effective information to supervise the keypoint sampling process. In terms of feature supervision, K is set to the number of channels of the multilayer perceptron input data. For the input feature matrix, the Euclidean distance of all eigenvalues ​​is calculated. Based on the set K value, the K nearest eigenvalues ​​are selected from the distance calculation results as representative features. Finally, the selected K values ​​are combined with the input feature point cloud as the basis for feature sampling.

[0130] Let x represent the point cloud input after candidate sampling, and p1 represent the feature map of the voxel convolution in this layer. The KNN method will complete the alignment and sampling of the input (x, p1) and perform normalization after maximum pooling to ensure that the specified number of points are input. The multilayer perceptron consists of two linear calculation layers and a fully connected layer. For this layer, the output of the previous layer (y, p2) is used for linear fusion calculation to meet the input requirements of the fully connected layer. Finally, this layer outputs the result (y1, p3) after network sampling. In this method, the output features are called key points and are used to supplement the features predicted by the second detection.

[0131] In order to achieve efficient global and local feature fusion, in this embodiment, preferably, the global feature and local feature fusion method uses region of interest (RoI) pooling to fuse global and local features, extract the target region of interest from the point cloud data, and improve the final detection accuracy.

[0132] RoI pooling is a convolutional neural network architecture that uses a global feature map as a foundation and performs feature fusion based on region proposals. It selects regions of interest, maps them to a voxel grid, and generates fixed-size feature vectors. These feature vectors are then used for subsequent object classification and regression tasks. The overall computational process is as follows:

[0133] (1) Based on the feature mapping matrix obtained by convolution of the original voxels, the voxel units corresponding to the candidate region are marked as the region of interest. The candidate region is adjusted to determine the position and size of the region of interest. Moreover, this method only performs feature fusion on the marked region of interest, so only the voxel units within the region of interest are marked as valid regions, while the remaining voxel units are marked as invalid regions.

[0134] (2) Divide the voxel units within the valid region into a fixed number of sub-regions. RoI pooling divides the voxel units within the valid region into a fixed number of sub-regions. This division can divide the region of interest into a fixed number of small blocks for parallel calculation, thereby improving subsequent calculation efficiency.

[0135] (3) Pooling is performed on the voxel units within each sub-region. For each sub-region, RoI pooling performs a pooling operation on the feature vectors of all voxel units within the sub-region. In this work, the maximum pooling method is used to fuse local point cloud features with global features, and finally obtain the feature vector of each sub-region.

[0136] (4) Connect the feature vectors of each sub-region according to the region of interest. RoI pooling connects the feature vectors of each sub-region to obtain the feature vector of the entire region of interest. This feature vector can be used as input for subsequent target classification and regression tasks.

[0137] The working principle and use process of the present invention:

[0138] First, global feature extraction based on the self-attention mechanism. This method leverages the attention mechanism within the Visual Transformer mechanism for feature extraction and learns features by correlating features of adjacent pixels to enhance the robustness of global features. Self-attention feature learning is integrated into the construction of a 3D feature extraction network and effectively combined with a convolutional neural network. This method retains more detailed target information in shallow layers and filters out peripheral features outside the target center, enabling the network to more effectively learn the overall feature information of the scene.

[0139] Then, based on the prior information, keypoint sampling is performed. This method divides the original point cloud space into four equal parts, parallelizes the calculation of the position of each point within the candidate region, directly discards points outside the candidate region, and only calculates the features of the keypoints within the candidate region. This processing method utilizes parallel computing acceleration and prior information parameter adjustment methods to effectively improve the sampling efficiency of keypoints while also preserving local target features, which is more conducive to the implementation of real-time feature extraction methods.

[0140] Finally, the global and local features are fused. The fusion method consists of two stages: the first stage provides prior information for extracting local features using global features, and the second stage integrates the local features into the global feature map in three-dimensional space. Throughout the entire process, a feature supervision mechanism is established to associate global and local features during the sampling process, thereby enhancing the robustness and generalization of the feature extraction network.

[0141] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A small object feature extraction method based on Visual-Transformer network, characterized in that: The following steps are included: Step 1: Global feature extraction based on self-attention mechanism: Use the attention mechanism in the Visual-Transformer mechanism to extract features, and learn features by associating the features of adjacent pixels. Integrate self-attention feature learning into the construction of the 3D feature extraction network, and effectively combine it with the convolutional neural network to retain more detailed target information in the shallow network and filter out peripheral features outside the target center. Step 2: Key point sampling based on prior information: By dividing the original point cloud space into four equal parts, the position of each point in the candidate area is calculated in parallel, and the points outside the candidate area are directly discarded, and only the features of the key points in the candidate area are calculated; Step 3: Fusion of global features and local features. The fusion method includes two stages: the first stage provides prior information for extracting local features using global features, and the second stage integrates local features into the global feature map in three-dimensional space.

2. According to claim 1, a small object feature extraction method based on Visual-Transformer network is characterized in that: The step 1 comprises: The point cloud data is divided into volume voxels in a regular 3D space to accommodate the input of the global feature extraction network, and the information of the point cloud is encoded and serialized through regularization processing; Based on the voxelization method, the voxel block size is established in the three-dimensional coordinate system, and the size is (r x , r y , r z ), for different data sets, different voxel block sizes are set according to the three-dimensional space distance of the scene to reduce the loss of feature points during processing; After determining the number of voxel blocks, each voxel block uses random sampling to collect 5 points of information to represent the voxel. If the number of point clouds is insufficient, repeated sampling is performed. At the same time, empty voxels are replaced with fixed values. In this way, after voxelization, a regularized three-dimensional point cloud feature matrix is ​​obtained.

3. The small object feature extraction method based on Visual-Transformer network according to claim 1 is characterized in that: The step 1 includes: the three-dimensional Transformer downsampling module adopts a sparse convolutional neural network as the key structure of the feature downsampling stage, adopts sparse matrices and compressed representations to store and process sparse weights, and uses sparse matrix-vector multiplication and specialized convolution operations to accelerate calculations, thereby completing feature extraction and fusion.

4. The small object feature extraction method based on Visual-Transformer network according to claim 3 is characterized in that: The step 1 comprises: The overall process of sparse convolutional neural network is: (1) Input data: Input voxelized point cloud data; (2) Sparse voxel coding: Sparse voxel hashing is used as the coding method for processing sparse voxels. The coding process is as follows: Store the position and feature values ​​of non-empty voxels in a hash table, where each hash table entry consists of a position key and a feature key. Where p is the three-dimensional position vector of the voxel, w is the random hash vector, s is the random offset, N is the size of the hash table, and H(p) represents the encoding result; Query the value h(p) of a specific voxel position to determine the position of the feature point in the voxel. h(p)=H(p) (2) The output of the code is V hashcode This involves retrieving the location and feature values ​​of all non-empty voxels in the hash table, and performing no operations on empty voxels. V hashcode =(p,f)∈H(p),p≠null (3) (3) Perform feature extraction using convolution operations: convolution is optimized for situations where there are only a few non-zero voxels; (4) Decoding encoded voxels: After feature extraction, the encoded voxels need to be decoded to restore the original point cloud representation. Formula (3) is used to complete the feature decoding operation and obtain the final feature map.

5. The small object feature extraction method based on Visual-Transformer network according to claim 4 is characterized in that: The step 1 comprises: The feature extraction network introduces a self-attention mechanism in the submanifold voxel module, which is implemented through parallel computing. In the attention mechanism, local attention and global attention are used simultaneously to learn the feature relations around the target area. The feature extraction network takes the feature map as input and consists of three network downsampling modules. Each sampling module contains a layer of convolutional neural network and two layers of sub-manifold voxel modules. The sub-manifold voxel module is the core structure of the feature extraction network. It converts point cloud data into voxel representation, which is a discrete three-dimensional voxel representation method. It also maps point cloud data to a low-dimensional space to reduce the dimensionality of point cloud data. The structure of the submanifold voxel module is self-attention processing and submanifold voxel convolution. It adopts a multi-head attention mechanism, including voxel attention and global attention. It distributes the calculation of related areas to different attention layers. Submanifold voxel convolution is used to process voxelized point clouds.

6. The small object feature extraction method based on Visual-Transformer network according to claim 5 is characterized in that: The calculation process of the self-attention mechanism is as follows: (1) Linear transformation of input data: For the input voxelized point cloud, three different linear transformation matrices are used, and the weight values ​​are (W Q , W K , W V ), map it to Query, Key, and Value to represent different target objects in the attention mechanism; (2) Linear data segmentation: The query, key, and value representations obtained from the input data are split and assigned to voxel attention and global attention. Each attention layer processes a part of the input sequence, allowing the model to learn different attention representations in parallel to capture the feature relationships between voxels and global features respectively. (3) Weight calculation: For each attention layer, use formula (4) to calculate the weight value to obtain the attention weight between each position in the voxel and other positions; Then, by weighting the input sequence with the attention weights, higher weights are used to highlight the voxel blocks that are more closely related to the current position in the input sequence. where F attention is the output feature of self-attention, N non-empty is the number of non-empty input voxelized point clouds. c , key embedding K c The sum value is embedded in V c It helps to extract the features C around the voxel center of the object region, where d is the voxel distance between two points; (4) Output results: The output information of the two attention layers is concatenated and mapped back to the original voxelized point cloud matrix to obtain the feature results calculated by the attention mechanism.

7. The small object feature extraction method based on Visual-Transformer network according to claim 5 is characterized in that: The sub-manifold voxel convolution calculation process is as follows: (1) Non-empty voxel selection: traverse the non-empty voxels in the voxelized point cloud data, and for each non-empty voxel, locate its position in the input data; (2) Convolutional feature sampling: Convolutional feature extraction is applied around the input data. The size and shape of the convolution kernel are consistent with the convolution kernel used in traditional convolution operations. In submanifold convolution, the convolution kernel is only calculated on non-empty voxels, and each element in the convolution kernel needs to be multiplied by its corresponding non-empty voxel. (3) Feature contribution calculation: After multiplying the elements in the convolution kernel by the corresponding non-empty voxels, the results are added to obtain the feature output value. This output value represents the new feature of the current non-empty voxel after the sub-manifold convolution operation. At the same time, the target center is set according to the annotation, and the feature contribution scores of adjacent non-empty voxels are calculated to filter out features far away from the target center and improve the representation of local detail features of the target. (4) Iterative calculation loop: Finally, iterate through all data, repeat feature calculation, and then determine the final global target feature output.

8. The small object feature extraction method based on Visual-Transformer network according to claim 3 is characterized in that: The sub-manifold voxel convolution calculation process is as follows: The three-dimensional Transformer downsampling network architecture consists of three network blocks, each of which contains a sparse convolutional neural network layer and two sub-manifold voxel modules. The sparse convolutional neural network realizes the function of downsampling the target. In the global feature extraction process, downsampling calculations are performed at three scales. At the same time, a sub-manifold voxel module is constructed for feature extraction at each sampling scale. The network layer data inside the sub-manifold voxel module is cross-layer connected to keep the feature weight sharing. The sub-manifold voxel module is used to redesign the non-empty voxel processing method. In this layer, the central non-empty voxel of each input data is determined, and the related non-empty voxel association relationship is calculated based on the set central voxel. The voxel feature attention module calculates the influence of the non-empty voxel on the target detection decision and obtains the confidence score of each voxel. The global feature attention module divides the entire image into n regions and calculates the correlation between each region and the global feature map. In addition, this layer only calculates the non-empty voxels in each region. Through the self-attention layer, the feature weight score of each non-empty voxel is obtained, and the confidence score of the non-empty voxel is calculated twice in the single-layer sub-manifold network.

9. The small object feature extraction method based on Visual-Transformer network according to claim 1, characterized in that: The step 2 comprises: Key point sampling uses a method of subdividing 3D point clouds and uses parallel computing to implement sampling processing. The original point cloud input data is spatially divided into 4 subsets, and key point sampling is performed on each subset separately. Parallel computing is used to accelerate key point sampling. Use the candidate box generated in the first stage as the supervision area, only sample the key points of the point cloud within the area, and do not calculate the points outside the area. The overall process of key point sampling includes two stages. The first stage is to divide the point cloud space, and the second stage is to sample key points based on the prior frame. The process is as follows: (1) Space division: Quarter space: Let S be the original point cloud space, and use K to represent the number of subdivided subspaces. The original point cloud space is divided into K equal parts, with K = 4, and the division is performed on the plane. Each subspace independently and in parallel calculates the key points in the area. The formula for space division is written as: Among them, S K is the result of spatial partitioning, K is the number of subdivided subsets, (arctan(py i , px i )∈(-π,π] is the angle of rotation along the positive direction of the X axis, (py i , px i ) is the final rotation angle, p i is the output of the point set obtained from the parallel computation; (2) Candidate point sampling: Generate candidate regions based on the detection method and perform key point sampling in a limited area. Let P be the candidate region and P′ be the key point information retained in the candidate region after screening. The calculation formula is as follows: where p i From formula (5), we can express the point cloud. The center area and scale size of each region are denoted as C and D respectively. j is the center point of region C, (dx j ,dy j ,dz j ) represents the length, width and height of the region, r (s) is the minimum radius of the region center point extending outward, P′ is the set of key points selected in the candidate region after point extraction, For the selection of key points in the region, GPU is used to accelerate the calculation. At the same time, according to the calculation process of step (1), the 3D space is divided into 4 subsets to perform the initial selection of key points.

10. The small object feature extraction method based on Visual-Transformer network according to claim 1, characterized in that: The step 3 comprises: In the process of key point feature extraction, the multi-layer perceptron uses 3 blocks to complete the feature downsampling process. Each block contains 2 layers, namely the feature map fusion layer and the multi-layer perceptron. The feature map fusion layer includes the K nearest neighbor layer and the maximum pooling layer. The multi-scale feature map in the three-dimensional Transformer downsampling network architecture is used as the supervision information for local key point sampling, and K nearest neighbors are used for feature selection to retain more effective information to supervise the key point sampling process. In terms of feature supervision, K is set to the number of channels of the multilayer perceptron input data. For the input feature matrix, the Euclidean distance of all eigenvalues ​​is calculated. According to the set K value, the K nearest eigenvalues ​​are selected from the distance calculation results as representative features. Finally, we combine the selected K values ​​with the input feature point cloud as the basis for feature sampling. x represents the point cloud input after candidate sampling, and p1 represents the feature map of voxel convolution in this layer. The KNN method will complete the alignment and sampling of the input (x, p1) and perform normalization after maximum pooling to ensure that the specified number of points are input. The multilayer perceptron consists of two linear calculation layers and one fully connected layer. For this layer, the output (y, p2) of the previous layer is used for linear fusion calculation to meet the input requirements of the fully connected layer. Finally, this layer outputs the result (y1, p3) after network sampling. In this method, the output features are called key points, which are used to supplement the features of the second detection prediction. The global feature and local feature fusion method uses the RoI pooling of the region of interest to fuse the global and local features and extract the target region of interest from the point cloud data. The overall calculation process of RoI pooling is as follows: (1) Based on the feature mapping matrix obtained by the original voxel convolution, the voxel units corresponding to the candidate region are marked as the region of interest, the candidate region is adjusted, the position and size of the region of interest are determined, and only the marked region of interest is subjected to feature fusion. Therefore, only the voxel units within the region of interest are marked as valid regions, and the remaining voxel units are marked as invalid regions; (2) Divide the voxel units in the effective area into a fixed number of sub-regions. RoI pooling divides the voxel units in the effective area into a fixed number of sub-regions; (3) Performing a pooling operation on the voxel units in each sub-region. For each sub-region, RoI pooling performs a pooling operation on the feature vectors of all voxel units in the sub-region. In this work, the maximum pooling method is used to fuse the local point cloud features with the global features, and finally obtain the feature vector of each sub-region; (4) According to the region of interest, the feature vectors of each sub-region are connected. The RoI pool connects the feature vectors of each sub-region to obtain the feature vector of the entire region of interest. This feature vector can be used as the input for subsequent target classification and regression tasks.

Images