Cylinder-voxel fusion-based airborne hyperspectral point cloud three-dimensional target detection system and method

Abstract

The invention discloses an airborne hyperspectral point cloud three-dimensional target detection system and method based on cylinder-voxel fusion, and belongs to the technical field of three-dimensional target detection. The objective of the invention is to solve the problems of difficulty in fully extracting features and mismatching of space and spectrum in existing three-dimensional target detection. The method comprises the following steps: firstly, through canopy density calculation and ground fitting, randomly adding an adversarial sample for a to-be-detected scene; then target detection is carried out based on an airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion, and the system realizes feature extraction of airborne hyperspectral point cloud and split geometric point cloud through a cylinder-voxel double-branch encoder; aligning, dynamically adjusting and integrating double-branch intermediate layer features through a sparse feature fusion module; and then adaptively aligning and selecting double-branch output features by using a block feature fusion module to obtain a dense aerial view feature map of the scene, and then inputting a detection head to obtain a prediction result.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional target detection, and in particular relates to a three-dimensional target detection system and method for airborne hyperspectral point clouds. Background Art

[0002] With the continuous development of 3D target detection technology, the use of point clouds for 3D target detection has important and extensive application value in the fields of intelligent transportation, urban planning, battlefield environment perception, etc. Airborne hyperspectral point clouds can simultaneously provide rich spectral information and spatial 3D information in the scene, and have higher application value in remote sensing scenes with rich ground object categories and similar distribution to the foreground and background. Existing 3D target detection algorithms are usually designed based on single-band point clouds, which makes it difficult to fully extract the complex spatial 3D-spectral features of airborne hyperspectral point clouds, and do not consider issues such as spatial and spectral mismatch. Therefore, the effect of 3D target detection based on point clouds needs to be further improved. Summary of the invention

[0003] The present invention aims to solve the problem of difficulty in fully extracting features and the problem of space and spectrum mismatch in existing three-dimensional target detection.

[0004] An airborne hyperspectral point cloud 3D target detection system based on cylinder-voxel fusion, comprising:

[0005] Hyperspectral point cloud data preprocessing unit: Split the three-dimensional position information from the hyperspectral point cloud to obtain the geometric point cloud; divide the space with the same horizontal grid size within the specified detection range to obtain the cylindrical hyperspectral point cloud and the voxelized geometric point cloud respectively:

[0006]

[0007] Among them, S v With S p Represent the spatial dimensions of the divided voxels and cylinders, and Represents the cylinder index and voxel index of the point cloud inside, N p With N v Respectively represent the number of cylinders and voxels with point clouds inside, [X p ,Y p ] and [X v ,Y v ,Z v ] represent the real coordinate sets of the cylinder and voxel with point cloud inside in the horizontal plane and space respectively;

[0008] Target detection unit: Target detection is performed based on hyperspectral point cloud and voxelized geometric point cloud. The specific process includes:

[0009] First, the cylindrical hyperspectral point cloud is input into the cylindrical branch, and the voxelized geometric point cloud is input into the voxel branch. At the same time, the sparse feature fusion module uses the cylindrical branch as an auxiliary to add the intermediate layer features of the two elements on the voxel branch.

[0010] The processing process of the column branch includes:

[0011] Use the point cloud encoder to integrate the absolute coordinates, normalized coordinates, center point, elevation range and spectral information of all point clouds in each cylinder to obtain the initial cylinder features. Among them C p Represents the feature dimension; then use the cylindrical encoder to extract features to obtain sparse features Then based on the sparse features The corresponding index The sparse features Convert to dense cylinder feature A tensor of dimension [H, W, C] is dense data;

[0012] The processing of the voxel branch includes:

[0013] Calculate the coordinate mean of the point cloud divided into the same voxel to obtain the initial voxel features Then use the voxel encoder to extract features to get sparse voxels Sparse voxel based The multi-scale feature aggregation module is used to obtain the sparse voxels F′ of the multi-scale receptive field v and the corresponding index I′ v ; Then based on the sparse voxel F′ v and the corresponding index I′ v , weight the sparse features in the vertical dimension, and then add all the features in the vertical direction to obtain the weighted compressed voxel features Finally Transformed into dense voxel features of dimension [H, W, C]

[0014] Then for the dense voxel features and dense column features Perform feature alignment and obtain the fused bird’s-eye view feature F through the multi-head attention mechanism bev ; Finally, F bev Input the detection head to get the prediction result.

[0015] Furthermore, the system further comprises a hyperspectral point cloud acquisition unit, and the hyperspectral point cloud acquisition unit is used to acquire hyperspectral point cloud data of the target detection area.

[0016] Furthermore, the process of integrating the absolute coordinates, normalized coordinates, center point, elevation range, and spectral information of all point clouds in each cylinder using a point cloud encoder includes:

[0017] The absolute coordinates, normalized coordinates, center point, elevation range and spectral information of all point clouds in the cylinder are concatenated along the feature dimension as input, features are extracted through a multi-layer perceptron, and then the maximum pooling operator is used to integrate the information along the dimension of the number of point clouds to obtain the initial cylinder feature F p .

[0018] Furthermore, the cylindrical encoder includes four convolution modules, the first convolution module includes two layers of submanifold sparse convolution, and the remaining convolution modules include multiple two-dimensional sparse convolution blocks; each two-dimensional sparse convolution block includes one layer of two-dimensional sparse convolution and two layers of submanifold sparse convolution; after the four convolution modules are processed, the sparse feature is output

[0019] Furthermore, the voxel encoder includes four convolution modules, the first convolution module includes two layers of submanifold sparse convolution and one dynamic focal convolution, and the remaining convolution modules include multiple three-dimensional sparse convolution blocks and one dynamic focal convolution; the three-dimensional sparse convolution block includes one layer of three-dimensional sparse convolution and two layers of submanifold sparse convolution; after the four convolution modules are processed, the sparse voxels are output.

[0020] Furthermore, based on sparse voxels The multi-scale feature aggregation module is used to obtain the sparse voxels F of the multi-scale receptive field. v ' and the corresponding index I v The process includes the following steps:

[0021] First, After two focal sparse convolution blocks in sequence, sparse voxels are obtained And the corresponding index Will According to the index And the corresponding sampling rate are added element by element to obtain the sparse voxel F that aggregates the multi-scale receptive field v ' and the corresponding index I v '.

[0022] Furthermore, based on the sparse voxel F v ' and the corresponding index I v ', weight the sparse features in the vertical dimension, and then add all the features in the vertical direction to obtain the weighted compressed voxel features The formula is as follows:

[0023]

[0024] Among them, Unique represents the set deduplication operator; index I v 'Stores the indexes of all features on the X, Y, and Z axes, I v '[:,(1,2)] refers to the index on the X and Y axes, Select index I through set deduplication operator v 'The non-repeated (x,y) values ​​in Represents index I v 'Distinct values ​​on the X and Y axes; F xy Represents the set of serial numbers of all features at position (x, y) on the XY plane. The total number of features is N v ', n is its serial number; F v '[q,:] represents all features at position (x,y) on the XY plane, I v '[q,3] represents the z-axis index of all features at position (x,y) on the XY plane, W[I v '[q,3] represents the elevation weights corresponding to all features at position (x,y) on the XY plane.

[0025] Furthermore, the sparse feature fusion module uses the cylinder branch as an auxiliary, and the process of adding the intermediate layer features of the two elements one by one on the voxel branch includes the following steps:

[0026] According to the position index I p with I v Achieve mutual alignment and calculate the associated index set I of the two p2v :

[0027]

[0028] in, Represents the index corresponding to the sparse cylinder and voxel, h i and w i Represents height and width respectively, x j ,y j and z j Represent the three-dimensional space coordinates respectively, and the calculated I p2v The correspondence between each voxel and the cylinder is stored; the size of the cylinder and the voxel is always consistent in each corresponding convolution module, and the cylinder gathers all the features at the corresponding plane position, so each voxel is aligned with a unique cylinder, according to the associated index set I p2v Constructing the correlation matrix

[0029]

[0030] M kl After multiplying with the cylinder feature, it is added to the voxel feature to achieve sparse fusion of cylinder and voxel features at corresponding positions:

[0031]

[0032] in, represents the features obtained by the two-dimensional sparse convolutional block in the cylindrical encoder, represents the features obtained by the focal sparse convolutional block in the voxel encoder, Indicates the features of the voxels after fusion.

[0033] Furthermore, for dense voxel features and dense column features Perform feature alignment and obtain the fused bird’s-eye view feature F through the multi-head attention mechanism bev The process is implemented by a block feature fusion module, and the block feature fusion module includes a feature alignment submodule and an adaptive channel selection submodule;

[0034] The feature alignment submodule first processes the dense voxel features and dense cylinder features:

[0035] Use a fully connected layer to encode each voxel feature into a query of dimension [M, 1, C], where M is the number of attention heads and C is the number of feature channels;

[0036] At the same time, position encoding is added to the corresponding K×K size neighborhood cylinder features and encoded into dimensions [M, C, K 2 ] keys and dimensions [M,K 2 ,C] value;

[0037] Through the multi-head attention mechanism, the outer product of the query and the key is calculated and normalized using the Softmax function to obtain a dimension of [M,1,K 2 ]; then, the matrix is ​​multiplied by the value and reconstructed to obtain a multi-head cross-attention feature with a dimension of [1, MC];

[0038] After the multi-head cross-attention features are compressed by the fully connected layer, the feature dimension becomes [1, C], and then concatenated with the original voxel features to finally obtain the alignment features of dimension [1, 2C];

[0039] After performing the above operations on all voxel features, the bird's-eye view alignment features are obtained:

[0040]

[0041] The adaptive channel selection submodule aligns the bird's-eye view features F align For further processing:

[0042] Taking each alignment feature as the center point, select the corresponding K×K size neighborhood features and add position encoding to obtain the dimension [K 2 ,C] feature blocks; use the fully connected layer to encode and reconstruct the feature blocks into dimensions [M,2C,K 2 ] query, value and dimension [M,K 2 ,2C]; through the multi-head attention mechanism, the outer product of the key and the query is calculated and normalized using the Softmax function to obtain an attention weight matrix with a dimension of [M,2C,2C]; then, the matrix is ​​multiplied by the value and reconstructed to obtain a matrix with a dimension of [M,2C,K 2 ]; after compression encoding by the fully connected layer, the fusion feature of dimension [1,2C] is obtained; the above operation is performed on all voxel features and concatenated, and finally the fusion bird's-eye view feature F is obtained bev .

[0043] A method for three-dimensional target detection based on cylinder-voxel fusion of airborne hyperspectral point cloud, firstly adding adversarial samples to the scene to be detected, and then using the airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion to perform target detection on the scene to be detected with the adversarial samples added;

[0044] The process of adding adversarial examples to the scene to be detected includes the following steps:

[0045] Randomly select the position O(x,y) in the scene as the center point, extract the neighboring plots within the specified radius, and query whether there is a point cloud with a height exceeding the specified height threshold in the plot; if not, record the position O in the list, otherwise do not record the position; repeat this process to obtain a list of the center points of the exposed plots; if the list is not empty, perform subsequent operations on each center point in the list:

[0046] First, set up iterations: randomly select samples from the adversarial sample library, read their bounding box labels, randomly rotate and move to the center point O(x, y) of a queried exposed land, and perform subsequent operations at each center point; in order to avoid overlap between the added adversarial samples and the real targets in the scene, calculate their intersection-over-union ratio based on the real target labels in the scene and the labels of the adversarial samples. If there is no overlap, the iteration ends. If there is overlap, repeat the random rotation and move to the center point of a queried exposed land and perform the above steps until a suitable adversarial sample is found or the iteration ends until the number of iterations is met;

[0047] If no suitable adversarial sample is found, the calculation of the current corresponding center point ends; if a suitable adversarial sample is found, all the point clouds below are removed according to their corresponding bounding box labels, the RANSAC algorithm is used to fit the ground elevation h of the area adjacent to the center point O(x, y), and then the adversarial sample is moved to the selected position (x, y, h), thereby adding the adversarial sample to the selected center point position.

[0048] Beneficial effects:

[0049] The purpose of the present invention is to use airborne hyperspectral point clouds to perform three-dimensional target detection to extract targets of interest in a scene, and to improve the existing three-dimensional target detection algorithm so that it can effectively extract heterogeneous spatial-spectral features and overcome the problem of spatial and spectral mismatch. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Add flowchart for adversarial examples.

[0051] Figure 2 Schematic diagram of the overall structure of the PiV-AHPC network.

[0052] Figure 3 Schematic diagram of the multi-scale aggregation module structure.

[0053] Figure 4 Schematic diagram of the block feature fusion module structure.

[0054] Figure 5 Visualization of predicted 3D bounding boxes for the HIT campus dataset, where red represents vehicles, green represents tents, and blue represents cartons.

[0055] Figure 6 This is a performance diagram of the structural migration ability analysis of the present invention. DETAILED DESCRIPTION

[0056] To this end, the first three-dimensional target detection algorithm PiV-AHPC focusing on airborne hyperspectral point clouds is designed in this paper. In order to extract complex spatial-spectral heterogeneous features and overcome spectral mismatch, a cylinder-voxel dual-branch encoder is designed to capture spectral, vertical complexity features and fine-grained spatial structure features respectively, realizing targeted mining of multi-source information; in order to strengthen branch association and overcome geometric distortion, a multi-level feature fusion mechanism is designed, and the branch intermediate layer association is established through the sparse fusion module, ensuring efficient interaction of information in spatial and channel dimensions; the block fusion module is used to adaptively select and align neighborhood features to improve the accuracy and richness of feature expression.

[0057] Specific implementation method 1: Combination Figure 2 To explain this embodiment,

[0058] This embodiment is a method for airborne hyperspectral point cloud 3D target detection based on cylinder-voxel fusion. It first randomly adds adversarial samples (corresponding to step 1), and then performs target detection based on an airborne hyperspectral point cloud 3D target detection system based on cylinder-voxel fusion (corresponding to steps 2-5). The specific process includes the following steps:

[0059] Step 1: Randomly add adversarial samples to the scene to be detected through canopy density calculation and ground fitting;

[0060] Step 2: Feature extraction of the airborne hyperspectral point cloud and the split geometric point cloud is realized respectively through the cylinder-voxel dual-branch encoder;

[0061] Step 3: Align, dynamically adjust and integrate the dual-branch intermediate layer features through the sparse feature fusion module;

[0062] Step 4: Adaptively align and select the output features of the two branches through the block feature fusion module to obtain a dense bird's-eye view feature map F of the scene bev ;

[0063] Step 5: F bev Input the detection head (3D object detector) to get the prediction result.

[0064] More specifically,

[0065] The specific process of step 1 is as follows:

[0066] In order to improve the algorithm's anti-interference ability and increase the difficulty of the task, the present invention designs a fake target generation algorithm called Fake-Paste, which is used to add adversarial samples to the scene. The adversarial sample here refers to an interference target that is similar to the real target in shape and structure, but has different spectral values, similar to the camouflage target used in the military. The process is as follows: Figure 1 Specifically,

[0067] Adversarial samples refer to interference targets that are similar in shape and structure to real targets but have different spectral values, similar to military camouflage targets. First, the target point cloud of interest is extracted from the training data according to the annotated 3D bounding box, randomly downsampled, and then background spectra (such as tree crowns, grass, etc.) are added to it to construct an adversarial sample library.

[0068] Since there are many key issues that have not been resolved in hyperspectral lidar, it is currently difficult to promote its application. In this case, a post-synthesis method is usually used to generate hyperspectral point cloud data. This method requires the use of two devices: lidar is used to obtain point cloud data, and spectrometer is used to obtain hyperspectral images. The synthesis process includes several steps: first, a digital surface model is created based on the point cloud data; then, this model is used to correct the hyperspectral image so that the two are aligned in spatial position; finally, the corresponding spectral information is assigned to each point cloud, and we use the nearest neighbor algorithm to find the corresponding spectral value. Although this post-synthesis method is relatively mature in technology and the operation process is more stable and reliable, it still has an obvious disadvantage: the spectral value of the occluded object in the point cloud may not be accurate enough. This is because hyperspectral images can only record the spectral information of surface objects, while lidar can penetrate the imaging to obtain the three-dimensional spatial structure of the occluded objects. Therefore, the spectral value of the occluded target in the hyperspectral point cloud may deviate from the actual situation. Although these occluded targets are real targets, they are very similar to the adversarial samples to be added. In order to avoid conflicts, the algorithm only adds adversarial samples in the exposed area. To this end, an iterative process for automatically querying exposed areas is designed:

[0069] Randomly select the location O(x,y) in the scene as the center point, extract the neighboring plots within the specified radius (the default is 3 meters), and query whether there is a point cloud with a height exceeding the specified height threshold in the plot. If not, record the location O in the list, otherwise do not record the location.

[0070] It should be noted that the point cloud data is normalized before being input into the network. The z value of all point clouds is subtracted from their minimum z value, so the height of the point cloud in the scene starts from 0. A height threshold is set, and the point clouds exceeding the height threshold are considered to be points that occlude the ground objects. The height threshold here is a priori set by humans.

[0071] The above iterative process is repeated to the specified number of iterations, and finally a list of the center points of the exposed land is obtained. If the list is empty, it means that there are too few exposed areas in the scene, and no adversarial samples are added; otherwise, the subsequent operations are performed on each center point in the list:

[0072] First do a fixed number of iterations:

[0073] Randomly select a sample from the adversarial sample library, read its bounding box label, randomly rotate it and move it to the center point O(x,y) of a queried exposed land, and perform subsequent operations at each center point. In order to avoid the added adversarial sample overlapping with the real target in the scene (which is not in line with the reality), calculate their intersection and union ratio based on the real target label (manually labeled) in the scene and the label of the adversarial sample. If there is no overlap, the iteration ends. If there is overlap, repeat the random rotation and move it to the center point of a queried exposed land and perform the above steps until a suitable (i.e., non-overlapping) adversarial sample is found or the iteration ends when the number of iterations is met.

[0074] If no suitable adversarial sample is found, the calculation of the current corresponding center point ends; if a suitable adversarial sample is found, all the point clouds below are removed according to its corresponding bounding box label to simulate the situation where there are no ground points below the real target due to occlusion.

[0075] Next, in order to accurately place the adversarial sample at the ground position corresponding to the center point O(x, y), the RANSAC algorithm is used to fit the ground elevation h of the area adjacent to the center point O(x, y):

[0076] First, take all point clouds whose distance from the center point O(x, y) on the XY plane is less than the distance threshold (set to 3 meters in this implementation) to form a point set A. It can be regarded as taking O as the center, demarcating a circular area with a specified radius (3 meters) on the XY plane, and then extending it to the Z axis to form a cylindrical space, and taking all point clouds in this space. Next, a fixed number of iterations are performed:

[0077] Each time, three non-collinear point clouds are randomly selected from point set A to construct a plane model, the distance between all point clouds and the plane is calculated, the points with a distance less than the division threshold are divided into in-plane points, and the points with a distance greater than the division threshold are divided into out-of-plane points, and the model parameters are recorded.

[0078] After the iteration, the plane model with the largest number of points in the plane is selected as the final fitting plane, that is, the ground plane.

[0079] The probability P of the RANSAC algorithm reaching a benign sampling subset can be expressed as:

[0080] P = 1-(1-(1-ε) 3 ) M

[0081] Among them, ε represents the error rate of the data, and M represents the number of iterations.

[0082] The elevation h of the ground plane is obtained according to the RANSAC algorithm, and then the adversarial sample is moved to the selected position (x, y, h), thereby adding the adversarial sample at the selected center point.

[0083] After completing the above operations for all the center points in the list, the adversarial sample addition to the scene to be detected is completed. Data representation, where N0 represents the number of point clouds, C h Represents the spectral dimension.

[0084] The present invention can better exert the performance of the model by adding adversarial samples, and can effectively improve the target recognition accuracy of the network model.

[0085] The specific process of step 2 is as follows:

[0086] S201. First, the input data (hyperspectral point cloud) must be split and voxelized and cylindricalized respectively:

[0087] The three-dimensional position information is separated from the hyperspectral point cloud to obtain the geometric point cloud. The space is divided into the same horizontal grid size within the specified detection range to obtain the cylindrical hyperspectral point cloud and the voxelized geometric point cloud respectively:

[0088]

[0089] Among them, S v With S p Represent the spatial dimensions of the divided voxels and cylinders, and Represents the cylinder index and voxel index of the point cloud inside, N p With N v Respectively represent the number of cylinders and voxels with point clouds inside, [X p ,Y p ] and [X v ,Y v ,Z v ] represent the real coordinate sets of the cylinder and voxel with point cloud inside in the horizontal plane and space respectively.

[0090] S202, next, the columnarized hyperspectral point cloud is input into the columnar branch, and the voxelized geometric point cloud is input into the voxel branch. Due to the vertical receptive field advantage of the column and the natural adaptability to the bird's-eye view, this branch extracts spectral features to assist classification and determines vertical structural features as spectral confidence criteria. Since the voxel structure retains spatial structural details, this branch captures fine-grained three-dimensional spatial features to achieve precise target positioning. Their respective structures are as follows:

[0091] (a) The column branch first needs to use a point cloud encoder to integrate the absolute coordinates, normalized coordinates, center point, elevation range, spectrum and other information in each column; the present invention takes the lightweight PointNet algorithm as an example, of course, other point cloud encoders can also be used. Specifically, the absolute coordinates, normalized coordinates, center point, elevation range and spectrum information of all point clouds in the column are first spliced ​​together along the feature dimension as input, and then features are further extracted through a multi-layer perceptron, and then the maximum pooling operator is used to integrate the information along the point cloud quantity dimension to obtain the initial column features. Among them C p Represents the feature dimension.

[0092] The cylindrical encoder is further used to extract features. The present invention takes the following process as an example: the encoder includes four convolution modules:

[0093] The first convolution module uses two layers of submanifold sparse convolution for preprocessing, and the remaining convolution modules are each processed using multiple two-dimensional sparse convolution blocks;

[0094] 2D sparse convolution block: A 2D sparse convolution layer is combined with two layers of submanifold sparse convolution to form a 2D sparse convolution block (SCB).

[0095] Ordinary sparse convolution operators downsample feature data, reducing computational load by focusing on relevant features and ignoring redundant spatial data. Submanifold convolutions maintain data sparsity and structural integrity by restricting the output to existing non-empty elements.

[0096] The cylindrical features and indices output by each convolution module are recorded as and Where i represents the i-th convolutional module.

[0097] After the four convolution modules are completed, the index of the sparse features based on the output The sparse features are downsampled by 8 times Convert to dense cylinder feature A tensor with dimensions [H, W, C] is dense data (data of dense cylindrical features). Since there is no data storage at many locations in three-dimensional data, in order to improve computing efficiency, sparse data is formed by recording the values ​​and indices of non-empty data. Dense data and sparse data can be converted into each other.

[0098] (b) The voxel branch first calculates the coordinate mean of the point cloud divided into the same voxel to obtain the initial voxel feature

[0099] The first half of the voxel branch is the voxel encoder (other advanced architectures can be used), which includes four convolution modules, each of which is a three-dimensional convolution block similar to the cylindrical branch, and a dynamic focal convolution kernel is introduced at the end of each convolution module, which can adaptively change the shape of the convolution kernel during training. In the last three convolution modules, the three-dimensional sparse convolution block and the dynamic focal convolution constitute a focal sparse convolution block (FSB), and the dynamic focal convolution kernel can retain more valuable foreground information.

[0100] In order to ensure that the intermediate layer features extracted by the dual branches consisting of the cylinder branch and the voxel branch have consistent resolution and similar level of abstraction in the horizontal direction, and to facilitate the subsequent element-by-element alignment and fusion of the features, the corresponding sparse convolution blocks and the focal sparse convolution blocks maintain the same convolution kernel size, step size, padding, and output feature dimension.

[0101] The symbolic definitions of voxel features and indices are similar to those of cylinders, namely and Where i represents the i-th convolutional module.

[0102] After four convolution modules, the sparse voxels with 8 times downsampling of the feature space are obtained.

[0103] In order to better overcome the spatial structural similarity between the target and the background in the airborne scene and the complex distribution of objects in space, the voxel branch is designed as the second half of the voxel branch. It provides a multi-scale 3D receptive field through an additional downsampling layer and efficiently integrates vertical features using weighted elevation compression. Its architecture is as follows: Figure 3 First, After two focal sparse convolution blocks, sparse voxels with 16 and 32 times downsampling are obtained. And the corresponding index and Compared to , they have a wider receptive field and lower spatial resolution. According to the index And the corresponding sampling rate is added element by element to obtain the sparse voxels that aggregate the multi-scale receptive field and index

[0104] When processing the voxel features output by the encoder, the traditional 3D object detection network usually directly connects the channels in series along the vertical direction and then compresses the features through the convolutional neural network. However, in the airborne scene, the vertical dimension of the detection area is large, and the use of the concatenation method will increase the consumption of computing resources. At the same time, the distribution of the objects of interest is obviously regional, and the effective information contained in the corresponding features of different elevation scenes is obviously different. Therefore, the design of the learnable parameter z∈R D×1, and further obtain the elevation weight W∈R D×1 , where D represents the number of vertical dimensions of the scene:

[0105]

[0106] Among them, z f 、z t is a learnable parameter.

[0107] By weighting the sparse features of each corresponding vertical dimension through W, and then summing all the features along the vertical direction, we can get the weighted compressed voxel features. With index The formula is as follows:

[0108]

[0109] Among them, Unique represents the set deduplication operator.

[0110] In fact, the index I v 'Stores the indexes of all features on the X, Y, and Z axes, I v '[:,(1,2)] refers to the index on the X and Y axes, Select index I through set deduplication operator v 'Non-repeated (x,y) values, that is, record all occurrences of (x,y), that is, Represents index I v 'Distinct values ​​on the X and Y axes; F xy Represents the set of serial numbers of all features at position (x, y) on the XY plane (there may be multiple features at different heights at this position), and the total number of features is N v ', n is its serial number. Therefore F v '[q,:] represents the position (x,y) on the XY plane (xy here is the recorded ) all the characteristics, I v '[q,3] represents the z-axis index of all features at position (x,y) on the XY plane (3 represents the z-axis), W[I v '[q,3] represents the elevation weights of all features at position (x,y) on the XY plane. In general, it is to sum the weighted features along the elevation dimension, turning three-dimensional data into a two-dimensional bird's-eye view.

[0111] Finally Transformed into dense voxel features of dimension [H, W, C]

[0112] The specific process of step 3 is as follows:

[0113] Unlike the traditional two-branch model that lacks information exchange during feature extraction, the present invention proposes a method for sparse feature fusion (SFF) in the middle layer. This method can establish information exchange between branches, which facilitates the network to dynamically adjust the feature correlation between branches. In addition, by fusing the spectral and vertical structural features extracted by the column branch into the voxel middle layer features, the subsequent region of interest pooling layer provides a spectral confidence criterion and a more effective classification basis, so that the category of the proposal can be better refined.

[0114] Specifically, the sparse feature fusion module uses the cylinder branch as an auxiliary to add the intermediate layer features of the two elements on the voxel branch. Since the feature maps of the convolution modules corresponding to the cylinder branch and the voxel branch always have the same spatial structure and feature dimension in the horizontal direction, the feature maps are summed up according to their respective position indexes I p with I v It can realize mutual alignment on the bird's-eye view, and then calculate the associated index set I of the two p2v :

[0115]

[0116] in, Represents the index corresponding to the sparse cylinder and voxel, h i and w i Represents height and width respectively, x j ,y j and z j Represent the three-dimensional space coordinates respectively, and the calculated I p2v The corresponding relationship between each voxel and the cylinder is stored. Since the size of the cylinder and the voxel is always consistent in each corresponding convolution module, and the cylinder gathers all the features at the corresponding plane position, each voxel is aligned with a unique cylinder. p2v The correlation matrix can be constructed

[0117]

[0118] M kl After multiplying with the cylinder feature and then adding it to the voxel feature, sparse fusion of cylinder and voxel features at corresponding positions can be achieved:

[0119]

[0120] in, Represents the features obtained by the two-dimensional sparse convolution block SCB in the cylindrical encoder, Represents the features obtained by the focal sparse convolution block FSB in the voxel encoder Indicates the features of the voxels after fusion.

[0121] The specific process of step 4 is as follows:

[0122] Dense voxel features based on the dual-branch output of step 3 and dense column features The two feature maps come from the same raw data and maintain similar structures during the encoding process, making the two feature maps consistent in space. However, due to the inherent mismatch problem of multi-sensor heterogeneous data and the unavoidable target registration error, spectral shift or assignment errors often occur. At the same time, learning the heterogeneous feature associations of the two branches and selecting important features in the feature dimension are key challenges to improving the algorithm's data expression capabilities. Simple concatenation or addition fusion methods are difficult to effectively solve these problems, so the following are designed: Figure 4 The block feature fusion module shown in Figure 1 can achieve accurate feature alignment and channel selection based on the learned modal association, and adaptively highlight more valuable features in the neighborhood.

[0123] The block feature fusion module includes a feature alignment submodule and an adaptive channel selection submodule.

[0124] In the feature alignment submodule, the dense voxel features and dense cylinder features are first processed. Specifically, the processing process includes:

[0125] Use a fully connected layer to encode each voxel (that is, a 1*1 voxel) feature into a query (Q) of dimension [M, 1, C], where M is the number of attention heads (default is 2) and C is the number of feature channels.

[0126] At the same time, position encoding is added to the corresponding K×K size neighborhood cylinder features (default K = 3), and a fully connected layer is used to encode them into dimensions [M, C, K 2 ] with keys (K) and dimensions [M,K 2 ,C] value (V).

[0127] Through the multi-head attention mechanism, the outer product of the query and the key is calculated and normalized using the Softmax function to obtain a dimension of [M,1,K 2 ] is the attention weight matrix. Subsequently, the matrix is ​​multiplied by the value and reconstructed to obtain the multi-head cross attention feature with dimension [1, MC].

[0128] After the multi-head cross-attention features are compressed by the fully connected layer, the feature dimension becomes [1, C], and then concatenated with the original voxel features to finally obtain the aligned features of dimension [1, 2C].

[0129] After performing the above operations on all voxel features, the bird’s-eye view alignment feature F is obtained. align ∈RH×W×2C , and its calculation formula is:

[0130]

[0131] In the adaptive channel selection submodule, the bird’s-eye view alignment feature F align This process implements the self-attention algorithm for the channel dimension, and its calculation process is similar to feature alignment. Specifically, its processing process includes:

[0132] Taking each alignment feature as the center point, select the corresponding K×K size neighborhood features (default K=3) and add position encoding to get the dimension [K 2 ,C] feature blocks. Use the fully connected layer to encode and reconstruct the feature blocks into dimensions [M,2C,K 2 ] query (Q), value (V), and dimension [M,K 2 ,2C] of the key (K). Through the multi-head attention mechanism, the outer product of the key and the query is calculated and normalized using the Softmax function to obtain an attention weight matrix with a dimension of [M,2C,2C]; then, the matrix is ​​multiplied with the value and reconstructed to obtain a matrix with a dimension of [M,2C,K 2 ] multi-head cross attention features. After compression encoding by the fully connected layer, the fusion feature of dimension [1,2C] is obtained. The above operation is performed on all voxel features and concatenated to finally obtain the fusion bird's-eye view feature F bev ∈R H×W×2C .

[0133] Through this block feature fusion module, the mismatch problem of multi-sensor heterogeneous data can be effectively handled, and accurate feature alignment and adaptive channel selection can be achieved, thereby improving the data expression ability and performance of the algorithm.

[0134] This method invented the first three-dimensional target detection algorithm for airborne hyperspectral point clouds, which can fully extract spatial-spectral heterogeneous features and overcome the problems of spectral mismatch and geometric distortion, thereby achieving accurate positioning and identification of targets of interest in the scene.

[0135] Example

[0136] In order to verify the performance of the algorithm proposed in the present invention, experiments were conducted on self-built data sets and public data sets, and the experimental results verified the effectiveness of the method proposed in the present invention. The following examples are used to verify the effect of the present invention:

[0137] 1. Experimental Dataset

[0138] Harbin Institute of Technology campus data set: A drone equipped with a Headwall-Nano hyperspectral camera and a VLP-16 lidar was used to collect scenes containing vehicles and carton targets on September 27, 2022. A drone equipped with the same hyperspectral camera and RIEGL-MiniVUX lidar was used to collect scenes containing vehicles and tent targets on July 7, 2023. The data covers the Harbin Institute of Technology Science Park and its surrounding areas, with a flight altitude of 80 meters. The total number of point clouds is 5349316 and 8797764, respectively, and the point cloud density is 84 and 169 (points / cubic meter), respectively. The hyperspectral spatial resolution is 0.8 meters, with a total of 273 bands, and the spectral range covers 400-1000 nanometers. After aligning the hyperspectral with the digital surface model generated by the point cloud, the hyperspectral point cloud is obtained by the approximate nearest neighbor algorithm. Due to hardware limitations, the data needs to be split into multiple small scenes to train the network. The data were split according to different flight strips and rotated at multiple angles. The sliding window method was used to crop the data along the bird's-eye view with a step size of 15 meters and a window size of 30 meters × 30 meters. Finally, 5421 sets of training scenes and 2654 sets of test scenes were obtained.

[0139] 2. Experimental Setup

[0140] 1) Evaluation indicators: Following the calculation method proposed by the public dataset Kitti, the detection accuracy of the 3D object detection algorithm is evaluated by calculating three evaluation indicators, namely, the average precision of the bird's-eye view (AP) BEV ), 3D average precision (AP 3D ) and average directional similarity (AOS). The above indicators can be combined with the precision and recall performance of the algorithm to comprehensively reflect the detection ability of the network. The average accuracy of the bird's-eye view, the average accuracy of three-dimensional space, and the average directional similarity calculate the average accuracy indicators from the bird's-eye view, three-dimensional space, and angle respectively. For categories of targets to be detected that are not sensitive to direction, such as tents and cartons, their average directional similarity indicators are calculated by the following formula to alleviate the direction penalty:

[0141]

[0142] Three difficulty levels are set: 'easy', 'medium', and 'difficult'. The threshold is set according to the number of point clouds of different categories of targets to achieve level division. The distribution of targets of different difficulty levels is as follows Figure 6 To evaluate the algorithm's ability to detect occluded targets, the target neighborhood scene is divided into three levels: "no occlusion", "partial occlusion", and "high occlusion" according to the canopy density of the target neighborhood scene.

[0143] 2) Implementation details: The algorithm is implemented using the code library OpenPCDet based on the pytorch framework. For both datasets, a distributed data parallel strategy is used to train the network on four TitanXP graphics cards. The batch size of a single card is set to 2, and the Adam optimizer is used. The learning rate and weight decay are both set to 0.01, and a total of 30 batches are trained. The input voxel size is set to (0.1m, 0.1m, 0.1m), the input cylinder size is set to (0.1m, 0.1m, 16m), and the range to be detected is set to (-25.6~25.6m, -25.6~25.6m, 0~16m).

[0144] 3. Results Analysis

[0145] 1) Comparative analysis with other methods

[0146] In order to prove the effectiveness of the PiV-AHPC algorithm, two representative algorithms of different solutions in the field of point cloud target detection were selected for quantitative and qualitative comparison on the HIT campus dataset, including PillarNet based on cylinders and VoxelNext based on voxel sparse prediction.

[0147] Table 1 shows the detection accuracy performance of the proposed method and the comparison algorithm on the HIT campus test set. According to its content, it can be found that PiV-AHPC shows the best detection performance on the HIT campus dataset. Compared with the two typical methods based on cylinders or voxels, the average accuracy indicators of the algorithm (especially in the detection of medium and difficult samples) have been significantly improved. This shows that the algorithm can ideally separate and detect the three targets of interest from the complex background with high interference, and the feature learning ability is better than using voxel or cylinder encoders alone, and can adapt to the distribution of data of different categories. For small and difficult-to-identify box targets, the algorithm's bird's-eye view detection accuracy at three difficulty levels is much higher than other methods. The main reason is that PiV-AHPC can make full use of the spatial and spectral information contained in the airborne hyperspectral point cloud, and can also perceive the category and position of the target based on the spectral features when the target point cloud is sparse. Figure 5 The detection results of these methods in large-scale scenes are shown, and it can be intuitively found that the proposed method can accurately classify and locate the objects of interest, showing strong anti-interference ability, and fewer false positives and false negatives.

[0148] Table 1 Performance analysis of 3D object detection methods on the HIT campus test set, the intersection-over-union thresholds of (vehicle, carton, tent) are (0.7, 0.5, 0.5)

[0149]

[0150]

[0151] 2) Comparative analysis under different canopy closures

[0152] Table 2 shows the target detection accuracy of PiV-AHPC in two data sets under different canopy density. There are no partially obstructed tents, unobstructed tents and boxes in the HIT campus data set. According to the table content, the detection accuracy of the algorithm is not highly correlated with the canopy density of the target area, and whether the target is obstructed has little effect on the PiV-AHPC detection performance. The above results illustrate the advantages of laser radars with penetration and three-dimensional imaging capabilities in detecting obstructed objects. On the other hand, the wide vertical receptive field of the PiV-AHPC column branches enables it to selectively use spectral information according to the specific target distribution and suppress the interference caused by erroneous spectra.

[0153] Table 2 Performance of PiV-AHPC under different occlusion levels on the HIT campus test set

[0154]

[0155] 3) Algorithm structure migration

[0156] In the above experiment, PiV-AHPC used the detection head proposed by Voxel-RCNN, but the core of the algorithm is a feature extraction scheme for hyperspectral point clouds, which can be freely migrated and applied to other 3D object detection architectures based on bird's-eye view prediction. This section takes CenterPoint based on center prediction and PV-RCNN++ based on point-voxel as examples to demonstrate the structural migration capability of the proposed method. The results are shown in Figure 2. Figure 6 As shown. Compared with the original network, the algorithm accuracy of the algorithm that has migrated the design structure of the present invention has been significantly improved in most detection indicators. The experimental results show the flexible structural migration capability of the voxel-column structure, which can be applied to many three-dimensional target detection architectures and effectively improve their detection capabilities for airborne hyperspectral point cloud data. In addition, the architecture is more applicable to the two-stage detection model because the rich information contained in the dual-branch intermediate features can be further released in the proposal refinement part, improving the details of target classification and positioning.

[0157] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. An airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion, characterized in that: include: Hyperspectral point cloud data preprocessing unit: Split the three-dimensional position information from the hyperspectral point cloud to obtain the geometric point cloud; divide the space with the same horizontal grid size within the specified detection range to obtain the cylindrical hyperspectral point cloud and the voxelized geometric point cloud respectively: Among them, S v With S p Represent the spatial dimensions of the divided voxels and cylinders, and Represents the cylinder index and voxel index of the point cloud inside, N p With N v Respectively represent the number of cylinders and voxels with point clouds inside, [X p ,Y p ] and [X v ,Y v ,Z v ] represent the real coordinate sets of the cylinder and voxel with point cloud inside in the horizontal plane and space respectively; Target detection unit: Target detection is performed based on hyperspectral point cloud and voxelized geometric point cloud. The specific process includes: First, the cylindrical hyperspectral point cloud is input into the cylindrical branch, and the voxelized geometric point cloud is input into the voxel branch. At the same time, the sparse feature fusion module uses the cylindrical branch as an auxiliary to add the intermediate layer features of the two elements on the voxel branch. The processing process of the column branch includes: Use the point cloud encoder to integrate the absolute coordinates, normalized coordinates, center point, elevation range and spectral information of all point clouds in each cylinder to obtain the initial cylinder features. Among them C p Represents the feature dimension; then use the cylindrical encoder to extract features to obtain sparse features Then based on the sparse features The corresponding index The sparse features Convert to dense cylinder feature A tensor of dimension [H, W, C] is dense data; The processing of the voxel branch includes: Calculate the coordinate mean of the point cloud divided into the same voxel to obtain the initial voxel feature Then use the voxel encoder to extract features to get sparse voxels Sparse voxel based The multi-scale feature aggregation module is used to obtain the sparse voxels F of the multi-scale receptive field. v ' and the corresponding index I v '; Then based on the sparse voxel F v ' and the corresponding index I v ', weight the sparse features in the vertical dimension, and then add all the features in the vertical direction to obtain the weighted compressed voxel features Finally Transformed into dense voxel features of dimension [H, W, C] Then for the dense voxel features and dense column features Perform feature alignment and obtain the fused bird’s-eye view feature F through the multi-head attention mechanism bev ; Finally, F bev Input the detection head to get the prediction result.

2. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 1, characterized in that: The system further comprises a hyperspectral point cloud acquisition unit, which is used to acquire hyperspectral point cloud data of the target detection area.

3. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 1, characterized in that: The process of using the point cloud encoder to integrate the absolute coordinates, normalized coordinates, center point, elevation range, and spectral information of all point clouds within each cylinder includes: The absolute coordinates, normalized coordinates, center point, elevation range and spectral information of all point clouds in the cylinder are concatenated along the feature dimension as input, features are extracted through a multi-layer perceptron, and then the maximum pooling operator is used to integrate the information along the dimension of the number of point clouds to obtain the initial cylinder feature F p .

4. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 3, characterized in that: The cylindrical encoder includes four convolution modules, the first convolution module includes two layers of sub-manifold sparse convolution, and the remaining convolution modules include multiple two-dimensional sparse convolution blocks; each two-dimensional sparse convolution block includes one layer of two-dimensional sparse convolution and two layers of sub-manifold sparse convolution; after the four convolution modules are processed, the sparse feature is output.

5. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 4, characterized in that: The voxel encoder includes four convolution modules, the first convolution module includes two layers of sub-manifold sparse convolution and one dynamic focus convolution, and the remaining convolution modules include multiple three-dimensional sparse convolution blocks and one dynamic focus convolution; the three-dimensional sparse convolution block includes one layer of three-dimensional sparse convolution and two layers of sub-manifold sparse convolution; after the four convolution modules are processed, the sparse voxels are output.

6. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to any one of claims 1 to 5, characterized in that: Sparse voxel based The multi-scale feature aggregation module is used to obtain the sparse voxels F of the multi-scale receptive field. v ' and the corresponding index I v The process includes the following steps: First, After two focal sparse convolution blocks in sequence, sparse voxels are obtained And the corresponding index Will According to the index And the corresponding sampling rate are added element by element to obtain the sparse voxel F that aggregates the multi-scale receptive field v ' and the corresponding index I v '.

7. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 6, characterized in that: Based on sparse voxel F v ' and the corresponding index I v ', weight the sparse features in the vertical dimension, and then add all the features in the vertical direction to obtain the weighted compressed voxel features The formula is as follows: Among them, Unique represents the set deduplication operator; index I v 'Stores the indexes of all features on the X, Y, and Z axes, I v '[:,(1,2)] refers to the index on the X and Y axes, Select index I through set deduplication operator v 'The non-repeated (x,y) values ​​in Represents index I v 'Distinct values ​​on the X and Y axes; F xy Represents the set of serial numbers of all features at position (x, y) on the XY plane. The total number of features is N v ', n is its serial number; F v '[q,:] represents all features at position (x,y) on the XY plane, I v '[q,3] represents the z-axis index of all features at position (x,y) on the XY plane, W[I v '[q,3] represents the elevation weights corresponding to all features at position (x,y) on the XY plane.

8. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 7, characterized in that: The sparse feature fusion module uses the cylinder branch as an auxiliary, and the process of adding the intermediate layer features of the two elements one by one on the voxel branch includes the following steps: According to the position index I p with I v Achieve mutual alignment and calculate the associated index set I of the two p2v : in, Represents the index corresponding to the sparse cylinder and voxel, h i and w i Represents height and width respectively, x j ,y j and z j Represent the three-dimensional space coordinates respectively, and the calculated I p2v The correspondence between each voxel and the cylinder is stored; the size of the cylinder and the voxel is always consistent in each corresponding convolution module, and the cylinder gathers all the features at the corresponding plane position, so each voxel is aligned with a unique cylinder, according to the associated index set I p2v Constructing the correlation matrix M kl After multiplying with the cylinder feature, it is added to the voxel feature to achieve sparse fusion of cylinder and voxel features at corresponding positions: in, represents the features obtained by the two-dimensional sparse convolutional block in the cylindrical encoder, represents the features obtained by the focal sparse convolutional block in the voxel encoder, Indicates the features of the voxels after fusion.

9. The airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion according to claim 8, characterized in that: For dense voxel features and dense column features Perform feature alignment and obtain the fused bird’s-eye view feature F through the multi-head attention mechanism bev The process is implemented by a block feature fusion module, and the block feature fusion module includes a feature alignment submodule and an adaptive channel selection submodule; The feature alignment submodule first processes the dense voxel features and dense cylinder features: Use a fully connected layer to encode each voxel feature into a query of dimension [M, 1, C], where M is the number of attention heads and C is the number of feature channels; At the same time, position encoding is added to the corresponding K×K size neighborhood cylinder features and encoded into dimensions [M, C, K 2 ] keys and dimensions [M,K 2 ,C] value; Through the multi-head attention mechanism, the outer product of the query and the key is calculated and normalized using the Softmax function to obtain a dimension of [M,1,K 2 ]; then, the matrix is ​​multiplied by the value and reconstructed to obtain a multi-head cross-attention feature with a dimension of [1, MC]; After the multi-head cross-attention features are compressed by the fully connected layer, the feature dimension becomes [1, C], and then concatenated with the original voxel features to finally obtain the alignment features of dimension [1, 2C]; After performing the above operations on all voxel features, the bird's-eye view alignment features are obtained: The adaptive channel selection submodule aligns the bird's-eye view features F align For further processing: Taking each alignment feature as the center point, select the corresponding K×K size neighborhood features and add position encoding to obtain the dimension [K 2 ,C] feature blocks; use the fully connected layer to encode and reconstruct the feature blocks into dimensions [M,2C,K 2 ] query, value and dimension [M,K 2 ,2C]; through the multi-head attention mechanism, the outer product of the key and the query is calculated and normalized using the Softmax function to obtain an attention weight matrix with a dimension of [M,2C,2C]; then, the matrix is ​​multiplied by the value and reconstructed to obtain a matrix with a dimension of [M,2C,K 2 ]’s multi-head cross-attention feature; After compression encoding by the fully connected layer, the fusion feature of dimension [1,2C] is obtained; the above operation is performed on all voxel features and concatenated, and finally the fusion bird's-eye view feature F is obtained. bev .

10. A method for airborne hyperspectral point cloud three-dimensional target detection based on cylinder-voxel fusion, characterized in that: First, an adversarial sample is added to the scene to be detected, and then, for the scene to be detected with the adversarial sample added, an airborne hyperspectral point cloud three-dimensional target detection system based on cylinder-voxel fusion as described in any one of claims 1 to 9 is used to perform target detection; The process of adding adversarial examples to the scene to be detected includes the following steps: Randomly select the position O(x,y) in the scene as the center point, extract the neighboring plots within the specified radius, and query whether there is a point cloud with a height exceeding the specified height threshold in the plot; if not, record the position O in the list, otherwise do not record the position; repeat this process to obtain a list of the center points of the exposed plots; if the list is not empty, perform subsequent operations on each center point in the list: First, set up iterations: randomly select samples from the adversarial sample library, read their bounding box labels, randomly rotate and move to the center point O(x, y) of a queried exposed land, and perform subsequent operations at each center point; in order to avoid overlap between the added adversarial samples and the real targets in the scene, calculate their intersection-over-union ratio based on the real target labels in the scene and the labels of the adversarial samples. If there is no overlap, the iteration ends. If there is overlap, repeat the random rotation and move to the center point of a queried exposed land and perform the above steps until a suitable adversarial sample is found or the iteration ends until the number of iterations is met; If no suitable adversarial sample is found, the calculation of the current corresponding center point ends; if a suitable adversarial sample is found, all the point clouds below are removed according to their corresponding bounding box labels, the RANSAC algorithm is used to fit the ground elevation h of the area adjacent to the center point O(x, y), and then the adversarial sample is moved to the selected position (x, y, h), thereby adding the adversarial sample to the selected center point position.

Images