A vehicle environment perception method based on multi-view and multi-scale feature fusion

Abstract

The present invention belongs to the technical field of vehicle environment perception, and specifically discloses a vehicle environment perception method based on multi-view and multi-scale feature fusion. This method aims at the problem of information loss in the projection process of the projection-based point cloud segmentation method, and builds a vehicle environment perception network based on multi-view and multi-scale feature fusion. In terms of network input, the present invention uses multi-layer distance view representation and bird's-eye view feature dual encoding to design a multi-level multi-projection fusion deep learning network, which effectively improves the information extraction accuracy of targets in point clouds with prominent vertical information and long distances, and reduces the information loss of separated projections. In terms of network structure, the present invention proposes a multi-view fusion module, which combines MFB and MMViT to calculate the high-order correlation between BEV features and RV features, and deeply fuses multi-view features. The present invention is conducive to avoiding losses incurred during the projection process and improving the vehicle environment perception capability.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving and relates to a vehicle environment perception method, and in particular to a vehicle environment perception method based on multi-view and multi-scale feature fusion. Background Art

[0002] LiDAR-based point cloud segmentation occupies a dominant position in the field of environmental perception and is widely used in various driving scenarios. Compared with vision-based segmentation models, LiDAR-based segmentation models are not affected by weather and lighting. At present, there are three main types of mainstream point cloud segmentation algorithms: the first is a point-based method that directly processes each point in the point cloud, such as PointNet, PointNet++, and RandLA-Net. By analyzing point cloud data point by point to capture spatial features, high-precision point cloud segmentation can be achieved; the second is a voxel-based method that divides the point cloud into a series of regular or irregular voxel units, and then performs three-dimensional convolution processing on them to extract spatial structure information, which is suitable for processing large-scale point cloud data; the third is a projection-based method that projects point cloud data onto a two-dimensional image and then processes it using two-dimensional convolution. This method can make full use of existing two-dimensional image processing technology, improve processing efficiency, and show good results in application scenarios.

[0003] The above three point cloud segmentation algorithms have the following problems: (1) Although the point-based method can effectively preserve the geometric information of environmental elements, the reasoning time is long due to the large number of points. (2) The voxel-based method faces the problem of computational redundancy because the uneven density distribution of the point cloud will cause many voxels to be empty. In addition, as the complexity of the point cloud increases, the efficiency of three-dimensional convolution will decrease. (3) The projection-based method uses top-down or spherical projection technology to convert point cloud data into bird's-eye view (BEV) and range view (RV) images. BEV projection can capture targets at different positions and is particularly suitable for processing targets with a large horizontal coverage range. However, this method will cause the loss of geometric information and height information, affecting the recognition of targets with more height information. RV projection can preserve the distance information of the original point cloud and capture more detailed data. However, due to the characteristics of point cloud scanning that objects are larger near and smaller far away, distant targets are often blocked in RV projection.

[0004] To solve these problems, current mainstream methods usually use a multi-projection fusion strategy. By combining projection data from different angles, the information loss caused by a single projection at the input or output stage is compensated. However, this does not completely or fundamentally solve the inherent information loss problem in the projection process, and there are also deficiencies in feature extraction. For example, in BEV projection, existing methods usually use multilayer perceptrons (MLP) and pooling operations to extract shallow features such as the average height of grid cells, but they cannot fully capture key geometric structure information. For RV projection, although current methods try to use BEV features to partially compensate for information, information loss due to projection occlusion is still an unsolved problem. Therefore, when the above-mentioned multi-projection fusion strategy is applied to the actual vehicle environment perception environment, it is easy to lead to poor vehicle environment perception capabilities, and thus fail to provide key support for safe and efficient decision-making and navigation of autonomous vehicles. Summary of the invention

[0005] The purpose of the present invention is to provide a vehicle environment perception method based on multi-view and multi-scale feature fusion. The method processes the original point cloud data based on dual encoding of bird's-eye view features and multi-layer distance view representation. At the same time, combined with the proposed vehicle environment perception network based on multi-view and multi-scale feature fusion, it is beneficial to solve the problem of information loss in the projection process of the projection-based point cloud segmentation method, thereby improving the vehicle environment perception ability.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A vehicle environment perception method based on multi-view and multi-scale feature fusion includes the following steps:

[0008] Step 1. After the collected original point cloud data is double-encoded with the bird's-eye view features, the bird's-eye view image, entropy value image and score image are output respectively; at the same time, the collected original point cloud data is represented by multi-layer distance views to obtain the overall RV image and multiple RV sub-images with different distances, and the importance of each different view is dynamically adjusted;

[0009] Step 2. Build a vehicle environment perception network based on multi-view and multi-scale feature fusion, which includes a range view encoding network, a bird's-eye view encoding network, a multi-view fusion module MVFM, and a detection head;

[0010] The bird's-eye view image, entropy image and score image are concatenated as the input of the encoder in the bird's-eye view encoding network;

[0011] The features of the extracted overall RV image are interactively connected with the features extracted from other RV sub-images, and finally the output is passed through a linear layer as the input of the encoder in the distance view coding network;

[0012] The distance view encoding network and the bird's-eye view encoding network are used to extract RV features and BEV features respectively. The two network structures are the same and both use the ResNet101 structure, which includes an encoder and a decoder;

[0013] The encoder includes four stages of encoding modules, which are respectively defined as the first, second, third and fourth stage encoding modules; the decoder includes four upsampling modules, which are respectively defined as the first, second, third and fourth upsampling modules;

[0014] An MVFM is inserted at the positions of the first and fourth stage encoding modules and the second upsampling module, respectively used to use the RV features and BEV features output by the corresponding stage networks of the distance view encoding network and the bird's-eye view encoding network as the input of the MVFM;

[0015] Perform deep fusion of multi-view features in MVFM; the RV features and BEV features after deep fusion and segmentation in each MVFM are returned to the distance view encoding network and the bird's-eye view encoding network respectively, and serve as the feature input of the next stage of the corresponding network;

[0016] The output of the decoder in the bird's eye view encoding network and the output of the decoder in the distance view encoding network are respectively converted from mesh to point cloud and then spliced. The spliced ​​features are input into the detection head, and the segmented point cloud result is output after detection.

[0017] Step 3. Use the image data obtained in step 1 to train the vehicle environment perception network built in step 2; for the input point cloud data of the current vehicle, use the trained vehicle environment perception network to obtain the vehicle environment perception result.

[0018] The present invention has the following advantages:

[0019] As described above, the present invention relates to a vehicle environment perception method based on multi-view and multi-scale feature fusion, which builds a vehicle environment perception network based on multi-view and multi-scale feature fusion. Among them, in terms of the input of the network structure, the present invention adopts multi-layer distance view representation and bird's-eye view feature dual encoding to design a multi-level multi-projection fusion deep learning network, which effectively improves the information extraction accuracy of targets in point clouds with prominent vertical information and far distances, and significantly reduces the information loss of separated projections. Specifically, the present invention generates multi-layer RV representations through projection to achieve selective attention to targets at different distances, thereby effectively suppressing the information loss caused by projection. In addition, multi-head attention is also used to adaptively adjust the weights of multi-layer RV inputs according to the correlation in different scenes, ensuring that the present invention can differentially encode significant elements in the environment, and multi-layer RV representation achieves selective attention to targets at different distances. BEV (bird's-eye view) encodes local height and geometric structure information through principal component analysis and normal distribution transform (Normal Distributions Transform, NDT), which greatly reduces the information loss caused by projection, thereby improving the overall understanding of the scene. The present invention effectively suppresses the loss of geometric structure information of point cloud BEV, uses MLP and pooling operations to extract preliminary features, combines non-destructive testing to extract local geometric structure information, and collaboratively realizes effective mining of BEV features. Multi-perspective input can effectively alleviate the limitations of single perspective and improve the robustness and anti-interference of the model. In terms of network structure, the present invention designs a distance view coding network, a bird's-eye view coding network, a multi-perspective fusion module MVFM and a detection head. In complex scenes or when there are perceptual anomalies (such as occlusion, dynamic objects, and sparse areas of point clouds), the network can flexibly utilize information from different perspectives and modalities to avoid decision-making errors caused by single-perspective input, thereby achieving more accurate vehicle perception. The MVFM proposed in the present invention combines the Multi-modal Factorized Bilinearpooling (MFB) with the Multiscale Multiview Vision Transformers (MMViT) to explore the high-order correlation between multi-view and multi-scale features, promote the deep fusion of different view features, and achieve information complementarity. The present invention deeply fuses multi-view features by calculating the high-order correlation between BEV features and RV features. The present invention helps to avoid the loss of existing point cloud perception methods during the projection process by improving the network structure and network input, retaining richer information, thereby improving the accuracy of target information extraction and the vehicle's environmental perception ability, thereby providing key support for safe and efficient decision-making and navigation of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an architecture diagram of a vehicle environment perception method based on multi-view and multi-scale feature fusion in an embodiment of the present invention.

[0021] Figure 2 is a flowchart of a multi-layer distance view representation module in an embodiment of the present invention;

[0022] Figure 3 1 is a flowchart of the processing of the dual encoding module of the bird's-eye view feature in an embodiment of the present invention;

[0023] Figure 4 4 is a flowchart of the processing of the multi-view fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0025] like Figure 1 As shown, this embodiment describes a vehicle environment perception method based on multi-view and multi-scale feature fusion, and the vehicle environment perception method based on multi-view and multi-scale feature fusion includes the following steps:

[0026] Step 1. After the collected original point cloud data is double-encoded with the bird's-eye view features, the bird's-eye view image, entropy image and score image are output respectively. At the same time, the collected original point cloud data is represented by multiple layers of distance views to obtain the overall RV image and multiple RV sub-images at different distances, and the importance of each different view is dynamically adjusted.

[0027] In this embodiment, the original point cloud data comes from the urban driving environment point cloud data disclosed on the Internet. The point cloud data is divided into several data sets, and the data sets are imported into the deep learning network structure to extract environmental information.

[0028] In this embodiment, two public data sets, SemanticKITTI and nuScenes, are used for training and verification. The data sets are divided according to the official data set division method. For example, SemanticKITTI has 00-21, a total of 22 sequence data. The 00-10 (except 08) sequence data is used as the training set, 08 as the verification set, and 11-21 as the test set.

[0029] Projecting a point cloud onto an RV image is a common method, however, projecting onto a single RV image often results in information loss, and is particularly prone to occlusion of distant targets. In addition, differences in the scene itself will also cause elements in different ranges to have different importance. Existing methods mainly focus on improving coding technology, and often ignore the variability of elements in different ranges of RV. In response to these problems, the present invention designs a multi-layer distance view representation module, which adopts a multi-layer projection method, that is, generating multiple RV sub-images to supplement the overall RV image, thereby providing a more comprehensive and effective input, aiming to better retain target features at different distances and heights, and extract more specific feature information. In addition, the present invention also introduces multi-head attention to dynamically adjust the importance of RV images in different ranges to obtain a more accurate representation of scene importance and improve the effectiveness of the perception process.

[0030] In the multi-layer distance view representation module, the point cloud data is first projected in multiple layers, and RV sub-images of different distances are generated according to different distance ranges to supplement the overall RV image, thereby providing a more comprehensive and effective input. Specifically, by presetting the distance interval of the RV projection, the projected point cloud subspace is divided in the original point cloud. The point cloud subspace is projected onto the RV sub-image. After the above processing, multiple RV sub-images of different distances are obtained, with a shape of 32×900.

[0031] like Figure 2 The processing flow chart of the multi-layer distance view representation module in this embodiment is shown. The multi-layer distance view representation module consists of two parts: multi-layer projection and adaptive weight adjustment. The multi-layer projection projects the point cloud data collected by the lidar into multiple distance view images according to different distances. The adaptive weight adjustment consists of convolution and multi-head attention.

[0032] In order to dynamically adjust the importance of different views, two convolutions with kernel size of 3×3, stride of 2×2 and no padding are first used to preliminarily extract the features of each RV image; then multi-head attention is used to map the input features to multiple subspaces and calculate the attention in parallel. The features extracted from the whole RV image are input into the multi-head attention as queries, and the features extracted from other RV sub-images are passed into the multi-head attention as keys and values ​​for feature interaction; in the multi-head attention, the input query, key and value first pass through multiple linear layers, and then enter the scaled dot product attention for interaction, and the interacted features are connected; finally, the interactive features are connected through the linear layer output as the input of the encoder in the distance view encoding network.

[0033] In the processing flow of multi-layer distance view representation, the projection formula is as follows:

[0034] ;

[0035] In the formula, is the distance measurement value of the kth point, represents the upper field of view of the LiDAR sensor, represents the vertical field of view of the lidar sensor, , Represents the lower field of view of the lidar sensor.

[0036] and Respectively represent the width and height of the distance view, is the pixel coordinate corresponding to the distance view of the kth point, Represents the coordinates of the kth point in the three-dimensional space in the original point cloud data.

[0037] The features of the extracted overall RV image are interactively connected with the features extracted from other RV sub-images, and finally output through a linear layer as the input of the encoder in the following distance view encoding network.

[0038] Height and geometric structure information are crucial in point cloud perception tasks. The current standard feature extraction method is to use MLP and pooling operations directly on the BEV mesh of the projected point cloud to extract and summarize surface features, such as the average position and average intensity of the points. However, these methods are not sufficient to capture the geometric structure of the point cloud, which is crucial for accurate perception.

[0039] In order to solve this limitation and extract object features more comprehensively, the present invention designs a bird's-eye view feature dual encoding module, such as Figure 3 As shown, the multi-layer perceptron MLP is combined with the normal distribution transform NDT to extract geometric structure information for each BEV grid. In addition, the intensity of the point can also provide rich information, so the point cloud intensity is processed separately to obtain the entropy value and score of the intensity. These geometric features are then connected with the features extracted by the MLP along the channel dimension. This rich set of features is used as the input of the encoder in the bird's-eye view encoding network, providing more comprehensive and detailed feature information for the present invention.

[0040] like Figure 3 As shown, the dual encoding module of the bird's-eye view feature in this embodiment is divided into two parts: one is the preliminary feature encoding part, and the other is the height and geometric information feature encoding part. The preliminary feature encoding part is composed of a multi-layer perceptron and a maximum pooling operation, and the height and geometric information feature encoding part is composed of principal component analysis and normal distribution transformation;

[0041] Firstly, the original point cloud input is rasterized in polar coordinates to adapt to the distribution characteristics of point cloud data, which is dense in the near field and sparse in the far field, and a point cloud represented by polar coordinate rasterization is obtained.

[0042] The preliminary feature encoding part uses a multi-layer perceptron to extract preliminary feature information such as average position and average intensity for each grid, and then uses the maximum pooling operation to condense the preliminary feature information to obtain a bird's-eye view image.

[0043] The height and geometric information feature encoding part first performs three-dimensional grid downsampling on the point cloud of each grid to reduce the density of the point cloud, and then uses principal component analysis to analyze the main directions of each point cloud feature and calculate the mean and covariance matrix of the point cloud.

[0044] The covariance matrix is ​​defined as follows:

[0045] ;

[0046] In the formula, is the covariance matrix of different dimensions of the point cloud, with a shape of 9×9, n is the feature dimension, and is defined as represents the covariance between the ith random variable and the jth random variable, .

[0047] Then, the Gaussian value is calculated using the mean and covariance matrix. Finally, based on these calculations, the entropy value and score of the point cloud and the reflection intensity of the point cloud are obtained, and an entropy image and a score image are drawn according to the entropy value and the score.

[0048] The entropy value reflects the structural complexity of the point cloud in the grid. The higher the entropy value, the more complex the structure. The score reflects the density of the point cloud in the grid. The higher the score, the more concentrated the point cloud, and the lower the score, the more dispersed the point cloud.

[0049] The entropy value and score effectively characterize the distribution and structure of objects, providing rich information support for point cloud perception and classification tasks.

[0050] The calculation formulas for the Gaussian value, entropy value, and score of the point cloud are as follows:

[0051] ;

[0052] ;

[0053] .

[0054] In the formula, represents the Gaussian value, represents the i-th point in the point cloud, represents the mean, represents the entropy value of the point cloud, In this embodiment, , represents the determinant, represents the score of the point cloud within the grid, represents the diagonal elements of the matrix, is an exponential operation.

[0055] Finally, the bird’s-eye view image, entropy image and score image are concatenated as the input of the encoder in the following bird’s-eye view encoding network.

[0056] Step 2. Build a vehicle environment perception network based on multi-view and multi-scale feature fusion, such as Figure 1 As shown, it includes a distance view encoding network, a bird's-eye view encoding network, a multi-view fusion module MVFM and a detection head.

[0057] The distance view encoding network and the bird's-eye view encoding network are used to extract RV features and BEV features respectively. The two network structures are the same and both use the ResNet101 structure, which includes an encoder and a decoder.

[0058] In this embodiment, the encoder includes a convolution block, a maximum pooling layer, and four-stage encoding modules.

[0059] The encoder is first downsampled by a convolutional block with a kernel size of 7×7 and a stride of 2, and then the spatial size of the feature map is further reduced by a max pooling layer with a kernel size of 3×3 and a stride of 2.

[0060] Subsequently, four stages of encoding modules are used for encoding. The encoding modules of each stage are composed of a different number of residual blocks, and each residual block is composed of two 1×1 convolutions and one 3×3 convolution.

[0061] Among them, the first stage encoding module includes 3 residual blocks, the second stage encoding module includes 4 residual blocks, the third stage encoding module includes 23 residual blocks, and the fourth stage encoding module includes 3 residual blocks.

[0062] Four stages of encoding modules are defined, which are the first, second, third and fourth stage encoding modules respectively. The features output by the maximum pooling layer pass through the first, second, third and fourth stage encoding modules in sequence.

[0063] The decoder includes four upsampling modules, which are defined as the first, second, third, and fourth upsampling modules. The features output by the encoder are gradually restored to size through the first, second, third, and fourth upsampling modules in sequence.

[0064] Each upsampling module in the decoder consists of deconvolution, skip connection, activation function and normalization; the skip connection in each upsampling module is used to realize the connection with the output of the corresponding size in the decoder.

[0065] in Figure 1The skip connections in are shown by the dotted arrows between the encoder and decoder.

[0066] Skip connections are used to pass feature maps from different stages in the encoder directly to the corresponding upsampling modules in the decoder, thereby preserving detail information and facilitating accurate reconstruction. Specifically, the output of a layer in the encoder can be connected to the corresponding upsampling layer in the decoder to enhance feature fusion and improve the segmentation performance of the network.

[0067] An MVFM is inserted at the position of the first and fourth stage encoding modules and the second upsampling module, which are used to use the RV features and BEV features output by the corresponding stage networks of the distance view encoding network and the bird's-eye view encoding network as the input of the MVFM.

[0068] Multi-view feature deep fusion is performed in MVFM; the RV features and BEV features after deep fusion and segmentation in each MVFM are returned to the distance view encoding network and the bird's-eye view encoding network respectively, and serve as the feature input of the next stage of the corresponding network.

[0069] Take the MVFM set in the first stage encoding module as an example:

[0070] The feature BEV output by the first-stage encoding module in the bird's-eye view encoding network and the feature RV output by the first-stage encoding module in the range view encoding network are used as the input of MVFM respectively, and multi-view feature deep fusion is performed.

[0071] After fusion, MVFM outputs RV features and BEV features that are deeply fused and segmented into two perspectives. The RV features are returned to the distance view coding network and serve as the input of the second-stage coding module of the distance view coding network.

[0072] Similarly, the BEV features are returned to the bird's-eye view encoding network and used as the input of the second-stage encoding module of the bird's-eye view encoding network.

[0073] The output of the decoder in the bird's-eye view encoding network and the output of the decoder in the distance view encoding network are respectively spliced ​​after mesh-to-point cloud G2P. The spliced ​​features are input into the detection head, and the segmented point cloud results are output after detection.

[0074] In the process of segmenting the point cloud, the result is an accurate classification of each point in the point cloud, classifying it into a specific category or object, such as ground, building, vehicle, pedestrian, tree, etc. This segmentation operation realizes the structured processing of point cloud data, enabling machines to understand and perceive the geometry and spatial distribution of the three-dimensional environment.

[0075] The segmented point cloud has an important correlation with vehicle environmental perception. As a key link in the autonomous driving system, the segmented point cloud provides rich three-dimensional spatial information for the environmental perception module.

[0076] Through the precise segmentation of point clouds, the present invention enables the vehicle to effectively identify and distinguish roads, obstacles, dynamic or static objects, and then perform operations such as accurate road detection, obstacle recognition and path planning.

[0077] Considering that the height information is retained when the point cloud is projected to RV, and the shape information is retained when it is projected to BEV, the present invention introduces a multi-view fusion module MVFM with an original design. MVFM encodes inputs from different perspectives and constructs a high-dimensional representation of feature inputs with multi-channel resolution, so that feature information from different perspectives and different resolutions can be effectively integrated to give play to the complementary advantages of multiple perspectives. In addition, the present invention uses a multi-modal decomposition bilinear pooling module MFB to replace the original linear layer, exploring high-order connections between different perspectives, thereby enhancing the representation ability of MVFM.

[0078] Unlike the traditional attention mechanism that only uses first-order feature interactions, the MVFM designed by the applicant realizes high-order feature interactions. It does not limit the resolution of RV features and BEV features, and can perform more comprehensive and in-depth multi-viewpoint fusion.

[0079] like Figure 4 As shown in FIG, the multi-view fusion module includes a multi-modal decomposition bilinear pooling module and a model framework MMViT. For a detailed introduction to MMViT, please refer to the following non-patent literature:

[0080] Liu Y, Ong N, Peng K, et al. Mmvit: Multiscale multiview visiontransformers[J]. arXiv preprint arXiv:2305.00104, 2023.

[0081] The processing flow of the multi-view fusion module MVFM in this embodiment is as follows:

[0082] The multi-view fusion module first performs patch embedding encoding and flattening operations on the RV features input by the distance view encoding network and the BEV features input by the bird's-eye view encoding network, and then divides them into query, key and value respectively.

[0083] The multimodal decomposition bilinear pooling module is used to calculate the high-order feature interactions between queries and keys and values ​​between different feature maps. The multimodal decomposition bilinear pooling module first decomposes the bilinear pooling matrix into two low-dimensional matrices by factorization, thereby reducing the number of parameters and improving storage efficiency.

[0084] A one-dimensional non-overlapping window size of K is used to sum and pool the low-dimensional features to merge the features, thereby capturing higher-order feature interactions and generating high-order features to fully respond to the complex relationships between the features.

[0085] In the multimodal decomposition bilinear pooling module, the calculation formula for sum pooling is as follows:

[0086] ;

[0087] In the formula, is a query with dimension m in the feature graph, is the key or value of dimension n in the feature graph, and is the projection matrix to be learned, is the potential dimension of the low-dimensional matrix after decomposition, It represents Hadamard product, i.e. element-wise multiplication, SumPooling represents sum pooling operation, and z is the high-order feature of the output.

[0088] After completing the multimodal decomposition bilinear pooling, the generated high-order features are concatenated with the input feature map of the multimodal decomposition bilinear pooling to form new keys and values, which are then input into MMViT to fully integrate the high-order features.

[0089] Finally, the fusion result is divided into two perspectives (i.e., RV and BEV features) through a segmentation operation to maintain the consistency and stability of the features. A complete and rich final feature representation is obtained through residual connection, which enhances the correlation between multi-perspective features.

[0090] Step 3. Use the image data obtained in step 1 to train the vehicle environment perception network built in step 2; for the input point cloud data of the current vehicle, use the trained vehicle environment perception network to obtain the vehicle environment perception result, that is, first input the point cloud data of the current vehicle collected by the on-board lidar sensor, and pre-process it using the method of step 1, and then input the pre-processing result into the trained vehicle environment perception network to obtain the vehicle environment perception result.

[0091] During the network training process, the original point cloud data comes from the urban driving environment point cloud data publicly available on the Internet. After being processed according to step 1, it is divided into several data sets, and the data sets are input into the deep learning network structure.

[0092] The training process of the vehicle environment perception network is as follows:

[0093] The random gradient descent method is used to iterate and optimize the network parameters. The vehicle environment perception network structure is pre-trained using the dataset constructed from the image data in step 1. The dataset is divided into a training set, a validation set, and a test set.

[0094] Specifically, the training set and validation set are used for network training, and the pre-training hyperparameters and loss function weights are set. The pre-training hyperparameters include the maximum number of iterations, initial learning rate, weight decay rate, and batch size.

[0095] After the vehicle environment perception network is trained, the network structure is tested using the test set to verify the effect of environmental information extraction.

[0096] First, we test it on the SemanticKITTI dataset. SemanticKITTI has 22 sequence data from 00 to 21, and uses sequence data from 00 to 10 (except 08) as the training set, 08 as the validation set, and 11 to 21 as the test set.

[0097] Use the trained deep learning network structure to predict the point cloud in the test set and use , , The indicators respectively evaluate the information extraction accuracy, quality and speed of the deep learning network structure.

[0098] , The definitions are as follows:

[0099] ;

[0100] .

[0101] In the formula, , and Represent the number of true positive points, false positive points and false negative points respectively, Represents category c , C represents the number of categories. At the same time, the commonly used neural networks SPVANS, Cylinder3D, PolarNet, GFNet, AMVNet and AF2S3Net in the field of point cloud information extraction are introduced to compare with the method of the present invention to verify the effectiveness of the network structure of the present invention. Table 1 shows the performance comparison of the method of the present invention and different neural networks on the SemanticKITTI dataset.

[0102] Table 1 Performance comparison of the proposed method and different neural networks on the SemanticKITTI dataset

[0103]

[0104] As shown in Table 1, compared with Cylinder3D, The improvement is 0.3%. This is mainly due to the combination of the multi-layer distance view representation module and the bird's-eye view feature dual encoding module proposed in the present invention, which minimizes the information loss in the projection process and realizes the effective information extraction of vertically extending objects such as telephone poles and fences and long-distance objects. In addition, since this embodiment is based on the projection method, it is less time-consuming. SalsaNext does not perform well in information extraction for long-distance roads or similar location targets, mainly because SalsaNext uses a four-channel BEV as input, which results in more serious information loss. Compared with Salsanext, this embodiment has This is attributed to the organic combination of multi-layer projection and adaptive weight adjustment in the multi-layer range view representation module, which effectively attenuates the information loss caused by target occlusion during RV projection. In addition, MVFM achieves complementary advantages between different perspectives, which improves the environmental information extraction effect of the present invention on such targets. Compared with GFNet, this embodiment An improvement of 2.4%. The geometric flow module used by GFNet cannot well mine the deep correlation between different perspectives. In contrast, the multi-perspective fusion module of this embodiment fully utilizes the high-order correlation between different perspective features by combining MFB and MMViT, and realizes deep fusion between different perspectives. The bird's-eye view feature dual encoding module very comprehensively extracts the preliminary features and geometric structure features of BEV, so that this embodiment provides richer information input.

[0105] Secondly, we tested it on the nuScenes dataset. We used the trained deep learning network structure to predict the point cloud in the dataset and used , , Indicators are used to evaluate the information extraction accuracy, quality and speed of this embodiment.

[0106] At the same time, the commonly used neural networks RandLA_Net, KPConv, SPVANS, Cylinder3D, PolarNet, SalsaNext, GFNet and AMVNet in the field of point cloud information extraction are introduced for comparison with this embodiment to further verify the effectiveness of the invention. Table 2 shows the performance comparison of this embodiment and different neural networks on the nuScenes dataset.

[0107] Table 2 Performance comparison of the proposed method and different neural networks on the nuScenes dataset

[0108]

[0109] As can be seen from Table 2, the table shows the environmental information extraction effect on the nuScenes dataset. This embodiment has a better effect on extracting targets with richer vertical information, which is mainly due to the effective combination of the multi-layer distance view representation module and the bird's-eye view feature dual encoding module adopted by the method of the present invention. Compared with the multi-projection fusion method GFNet, this embodiment has a better effect on extracting targets with richer vertical information. The improvement is 2.4%, which is mainly attributed to the multi-view fusion module adopted by the present invention, which mines the high-order correlation between different views and realizes deeper feature fusion. This embodiment has a satisfactory information extraction effect for targets with richer vertical information on the road surface farther away.

[0110] In addition, in order to further verify the role of the multi-layer distance view representation module and the bird's-eye view feature dual encoding module in the deep learning network structure, the network containing only the backbone encoding network, the network with the multi-layer distance view representation module, the network with the bird's-eye view feature dual encoding module, and the network with the multi-view fusion module were trained on the SemanticKITTI dataset and used The environmental information extraction accuracy of each network is evaluated by using the indicators, and the verification results are shown in Table 3.

[0111] Table 3 Environmental information extraction accuracy of different deep learning network structures

[0112]

[0113] As can be seen from Table 3 above, the multi-layer distance view representation module projects the point cloud onto multiple RVs, which avoids occlusion loss while focusing on targets at different distances. The improvement is 1.5%, proving the effectiveness of the multi-layer distance view representation module. The bird’s-eye view feature dual encoding module uses NDT to fully capture the information of the target’s geometric structure, thereby providing richer input information and converting The multi-view fusion module uses the clever combination of MFB and MMViT to explore the high-order correlation between different perspectives and achieve deeper fusion, thereby improving performance ( increased by 0.7%).

[0114] The method of the present invention can effectively reduce the information loss in the projection process, realize deeper information complementarity between RV and BEV, achieve more efficient and accurate point cloud information extraction effect, improve the target information extraction accuracy, thereby improving the vehicle environment perception ability, and further provide key support for safe and efficient decision-making and navigation of autonomous driving vehicles.

[0115] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A vehicle environment perception method based on multi-view and multi-scale feature fusion, characterized in that: The steps include: Step 1. After the collected original point cloud data is double-encoded with the bird's-eye view features, the bird's-eye view image, entropy value image and score image are output respectively; at the same time, the collected original point cloud data is represented by multi-layer distance views to obtain the overall RV image and multiple RV sub-images with different distances, and the importance of each different view is dynamically adjusted; Step 2. Build a vehicle environment perception network based on multi-view and multi-scale feature fusion, which includes a range view encoding network, a bird's-eye view encoding network, a multi-view fusion module MVFM, and a detection head; The bird's-eye view image, entropy image and score image are concatenated as the input of the encoder in the bird's-eye view encoding network; The features of the extracted overall RV image are interactively connected with the features extracted from other RV sub-images, and finally the output is passed through a linear layer as the input of the encoder in the distance view coding network; The distance view encoding network and the bird's-eye view encoding network are used to extract RV features and BEV features respectively. The two network structures are the same and both use the ResNet101 structure, which includes an encoder and a decoder; The encoder includes four stages of encoding modules, which are defined as the first, second, third, and fourth stage encoding modules respectively; The decoder includes four upsampling modules, which are respectively defined as the first, second, third and fourth upsampling modules; An MVFM is inserted at the positions of the first and fourth stage encoding modules and the second upsampling module, respectively used to use the RV features and BEV features output by the corresponding stage networks of the distance view encoding network and the bird's-eye view encoding network as the input of the MVFM; Perform deep fusion of multi-view features in MVFM; the RV features and BEV features after deep fusion and segmentation in each MVFM are returned to the distance view encoding network and the bird's-eye view encoding network respectively, and serve as the feature input of the next stage of the corresponding network; The output of the decoder in the bird's eye view encoding network and the output of the decoder in the range view encoding network are respectively converted from mesh to point cloud and then spliced. The spliced ​​features are input into the detection head, and the segmented point cloud result is output after detection. The multi-view fusion module includes a multi-modal decomposition bilinear pooling module and a model framework MMViT; First, the RV features of the distance view encoding network input and the BEV features of the bird's eye view encoding network input are patch-embedded and flattened, and then divided into query, key and value respectively; The multimodal decomposition bilinear pooling module is used to calculate the high-order feature interactions between queries and keys and values ​​between different feature graphs. The multimodal decomposition bilinear pooling module first decomposes the bilinear pooling matrix into two low-dimensional matrices by factorization, and performs a one-dimensional non-overlapping window size K sum pooling operation on the low-dimensional features to generate high-order features; After completing the multimodal decomposition bilinear pooling, the generated high-order features are concatenated with the input feature map of the multimodal decomposition bilinear pooling to form new keys and values, which are then input into MMViT to fully integrate the high-order features. Finally, the fusion result is divided into two perspectives through segmentation operation, and a complete and rich final feature representation is obtained through residual connection; Step 3. Use the image data obtained in step 1 to train the vehicle environment perception network built in step 2; for the input point cloud data of the current vehicle, use the trained vehicle environment perception network to obtain the vehicle environment perception result.

2. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In step 1, the processing flow of the double encoding of the bird's-eye view features is as follows: First, the input original point cloud data is rasterized in polar coordinates to obtain a point cloud represented by polar coordinate rasterization; Secondly, the preliminary feature encoding part uses a multi-layer perceptron to extract preliminary feature information including average position and average intensity for each grid, and then uses the maximum pooling operation to condense the preliminary feature information to obtain a bird's-eye view image; The height and geometric information feature encoding part first downsamples the point cloud of each grid, then uses principal component analysis to analyze the main direction of each point cloud feature, and uses normal distribution transformation to calculate the entropy value and score of the point cloud and the reflection intensity of the point cloud, and draws the entropy value image and score image according to the entropy value and score respectively; Finally, the bird's-eye view image, entropy image and score image are concatenated as the input of the encoder in the bird's-eye view encoding network.

3. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 2 is characterized in that: In step 1, principal component analysis is used to analyze the main directions of each point cloud feature, and the mean and covariance matrix of the point cloud are calculated; the covariance matrix is ​​defined as follows: Where cov_matrix is ​​the covariance matrix of different dimensions of the point cloud, with a shape of 9×9, n is the feature dimension, and q is defined as ij represents the covariance between the i-th random variable and the j-th random variable, i,j=1,2,...,n; Then, the Gaussian value is calculated using the mean and covariance matrix; finally, the entropy value and score of the point cloud and the reflection intensity of the point cloud are obtained; wherein the calculation formulas of the Gaussian value, entropy value and score are as follows: score=∑exp(diag(gaussianValue)); Where, gaussianValue represents the Gaussian value, p i represents the i-th point in the point cloud, μ represents the mean, entropy represents the entropy of the point cloud, N represents the dimension of the point cloud feature, det represents the determinant, score represents the score of the point cloud in the grid, diag represents the diagonal elements of the matrix, and exp represents the exponential operation.

4. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In step 1, the processing flow of multi-layer distance view representation is as follows: The original point cloud data is first projected in multiple layers to generate RV sub-images of different distances according to the different distance ranges to supplement the overall RV image. Then, two convolutions with a kernel size of 3×3, a stride of 2×2 and no padding are used to preliminarily extract the features of each RV image. Then, multi-head attention is used to map the input features into multiple subspaces and calculate the attention in parallel; Specifically, the features extracted from the whole RV image are input into the multi-head attention as queries, and the features extracted from other RV sub-images are passed into the multi-head attention as keys and values ​​for feature interaction; In multi-head attention, the input query, key, and value first pass through multiple linear layers, and then are input into the scaled dot product attention for interaction, and the features after interaction are connected; finally, the features after interaction are connected through the linear layer output as the input of the encoder in the distance view encoding network.

5. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 4 is characterized in that: In the processing flow of the multi-layer distance view representation, the projection formula is as follows: In the formula, r k is the distance measurement value of the kth point, f up represents the upper field of view of the LiDAR sensor, f represents the vertical field of view of the LiDAR sensor, and f = f up +f down , f down represents the lower field of view of the LiDAR sensor; w and h represent the width and height of the distance view, respectively. is the pixel coordinate corresponding to the distance view of the kth point, (x k ,y k ,z k ) represents the coordinates of the kth point in the three-dimensional space in the original point cloud data.

6. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In step 2, the encoder includes a convolution block, a maximum pooling layer, and four-stage encoding modules; The encoder first downsamples by a convolutional block with a kernel size of 7×7 and a stride of 2, and then further reduces the spatial size of the feature map through a maximum pooling layer with a kernel size of 3×3 and a stride of 2; Then, four stages of encoding modules are used for encoding. Each stage of encoding modules consists of a different number of residual blocks, and each residual block consists of two 1×1 convolutions and one 3×3 convolution. Among them, the first stage encoding module includes 3 residual blocks, the second stage encoding module includes 4 residual blocks, the third stage encoding module includes 23 residual blocks, and the fourth stage encoding module includes 3 residual blocks; Each upsampling module in the decoder consists of deconvolution, skip connection, activation function and normalization; the skip connection in each upsampling module is used to realize the connection with the output of the corresponding size in the decoder.

7. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In the multimodal decomposition bilinear pooling module, the calculation formula for summing pooling is as follows: Where x is the query with dimension m in the feature graph, y is the key or value with dimension n in the feature graph, and is the projection matrix to be learned, k′ is the potential dimension of the low-dimensional matrix after decomposition, It represents Hadamard product, i.e. element-wise multiplication, SumPooling represents sum pooling operation, and z is the high-order feature of the output.

8. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In step 3, the training process of the vehicle environment perception network is as follows: Iterate and optimize network parameters through the stochastic gradient descent method, and use the dataset constructed from the image data in step 1 to pre-train the vehicle environment perception network structure. The dataset is divided into a training set, a validation set, and a test set. Specifically, the network is trained using the training set and validation set, and pre-training hyperparameters and loss function weights are set. The pre-training hyperparameters include the maximum number of iterations, initial learning rate, weight decay rate, and batch size. After the vehicle environment perception network is trained, the network structure is tested using the test set to verify the effect of environmental information extraction.

9. The vehicle environment perception method based on multi-view and multi-scale feature fusion according to claim 1 is characterized in that: In step 3, the process of obtaining the vehicle environment perception result by using the trained vehicle environment perception network is as follows: First, input the point cloud data of the current vehicle collected by the lidar sensor and preprocess it using the method in step 1. Then input the preprocessed result into the trained vehicle environment perception network to obtain the vehicle environment perception result.

Images