Automatic driving vehicle environment sensing method for enhancing small target detection

Abstract

The invention provides an automatic driving vehicle environment perception method for enhancing small target detection based on a visual sensor, and relates to the technical field of target detection of automatic driving vehicles, and the detection method comprises the following steps: obtaining an omnibearing image around a vehicle through a multi-angle visual sensor; performing multi-scale feature extraction on the images at different angles; the multi-scale features are projected to a BEV (Bird's Eye View, aerial view) space; designing a detection head, and decoupling the category and position information of an object in the input feature map; and evaluating and optimizing the model. According to the method, a small target detection method and a BEV sensing algorithm are combined, and the multi-angle camera data and the neural network are utilized, so that the vehicle can obtain surrounding object sensing with the vehicle as the center, and data support is provided for subsequent positioning, path planning and vehicle control.

Description

Technical Field

[0001] The present invention relates to the field of environmental perception technology for autonomous driving vehicles based on visual sensors, and in particular to a small target visual perception algorithm for autonomous driving vehicles. Background Art

[0002] Three-dimensional spatial perception plays a key role in the development of autonomous driving technology. Although lidar-based methods have made significant progress, visual sensor-based methods have received more attention. Compared with lidar methods, visual sensor-based methods are not only less expensive, but also show encouraging advantages in identifying road elements and other aspects. Bird's Eye View (BEV), as a unified representation of the surrounding environment, clearly shows the location and size of objects. It is used in various tasks of autonomous driving, such as perception and planning. The core concept of BEV involves projecting the 3D environment data around the vehicle onto the top view plane to generate a 2D bird's-eye view. 3D object detection algorithms based on depth information require accurate BEV features to support the prediction of 3D bounding boxes.

[0003] Due to the rapid changes in autonomous driving scenes, small objects in 3D detection tasks may not generate corresponding depth distributions due to sparse features or occlusion. Inaccurate depth distributions can lead to the loss of small objects in the BEV (bird's eye view) features, seriously affecting the overall detection performance and increasing potential safety risks in autonomous driving. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the prior art and propose an environment perception method for autonomous driving vehicles with enhanced small target detection. The present invention designs an enhanced small target detection network based on visual sensors and develops a projection network based on the fusion of global features and local features.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for autonomous driving vehicle environment perception with enhanced small target detection, characterized in that it comprises the following steps:

[0007] S1: Obtain all-round images around the vehicle through multi-angle visual sensors;

[0008] S2: Extract multi-scale features from images at different angles;

[0009] S3: Project the multi-scale features into the BEV (Bird's Eye View) space;

[0010] S4: Design a detection head to decouple the category and location information of objects in the input feature map;

[0011] S5: Evaluate and optimize the model.

[0012] Preferably, in S1, six cameras are used to obtain images at different angles centered on the autonomous driving vehicle, using png as the data format.

[0013] Preferably, S2 includes a process of acquiring multi-scale features and fusing spatiotemporal features.

[0014] Preferably, the specific steps in S2 are as follows:

[0015] Firstly, a multi-scale feature acquisition model is established, which consists of a two-dimensional backbone network ResNet for basic feature extraction and a neck FPN for multi-scale variable objects;

[0016] In the model, for each input image of shape H×W×3, after passing through the multi-scale feature acquisition model, a Multi-scale features P2, P3, P4, P5.

[0017] In the above formula, H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image;

[0018] Then, a spatiotemporal feature fusion process is performed: the spatiotemporal feature fusion process is performed using a convolutional neural network;

[0019] The spatiotemporal fusion process is expressed as follows:

[0020] F=(P2,P3,P4,P5)

[0021] W t =Sigmoid(Conv(F))

[0022] W′ t =αW t +βW′ t-1

[0023] F weighted =F⊙W′ t

[0024] F i =Conv(Concat(F weighted ))

[0025] Where Conv(·) represents the convolution operation, Sigmoid(·) represents the Sigmoid activation function, α and β are weight factors, ⊙ represents element-by-element multiplication, and Concat(·) represents the feature map cascade operation.

[0026] Preferably, S3 includes image feature point cloud generation and BEV feature generation.

[0027] Preferably, the specific steps in S3 are as follows:

[0028] Firstly, a model for generating image feature point clouds is established. The coordinates of the generated view cone point clouds in the image coordinate system and the corresponding depth values ​​are described by (u, v, d). The Efficientnet-B0 backbone network is used to extract features from the surround view image.

[0029] In the model,

[0030]

[0031] Among them, K represents the camera internal parameter, X cam , Y cam , Z cam is the 3D coordinate in the camera coordinate system, X ego , Y ego , Z ego is the 3D coordinate in the ego-vehicle coordinate system, (R, t) is the camera external parameter;

[0032] Then, BEV feature generation is performed: the feature generation process uses global pooling and local pooling, which is expressed as follows:

[0033]

[0034] F loacl (c) = Conv{concat[f local (i,j,c)]}

[0035]

[0036] Where F(i,j,c) is the feature extracted by Efficientnet-B0, F global (c) is the feature obtained by global pooling, F local (c) is the feature obtained by local pooling, F BEV is the generated BEV feature, i and j represent the row and column indices of the feature map, H and W represent the height and width of the feature map, c represents the channel index, and k represents the pooling window size.

[0037] Preferably, S5 includes parameter tuning and iterative optimization.

[0038] Preferably, the parameter tuning includes performing parameter tuning using an automatic parameter tuning tool or algorithm;

[0039] The iterative optimization: further improve and optimize the model based on the evaluation results and feedback from parameter tuning.

[0040] Compared with the prior art, this application has the following beneficial effects:

[0041] 1. The present invention designs an enhanced small target detection network based on visual sensors, which combines the small target detection method and the BEV perception algorithm, and uses multi-angle camera data and neural networks to enable the vehicle to obtain the perception of surrounding objects centered on itself, thereby providing data support for subsequent positioning, path planning and vehicle control.

[0042] 2. The present invention designs a feature projection network, which rotates the features in the camera coordinate system to the vehicle coordinate system through a rotation matrix, and then performs global pooling and local pooling on the multi-scale features to construct small target features. The Transformer is used to summarize the overall background and context information of the global features and the small target information of the local features, and project them into the BEV space. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flowchart of the environment perception method for autonomous driving vehicles with enhanced small target detection based on visual sensors.

[0044] Figure 2 Structural diagram for multi-scale feature extraction.

[0045] Figure 3 The network structure diagram of multi-scale features projected into the BEV space.

[0046] Figure 4 This is the network structure flow chart of the detection head. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with specific embodiments.

[0048] See also Figure 1 , an environment perception method for autonomous driving vehicles with enhanced small target detection based on visual sensors, comprising the following steps:

[0049] S1: Obtain all-round images around the vehicle through multi-angle visual sensors;

[0050] The multi-angle visual sensor includes: 6 cameras at different angles and internal and external parameter calibration;

[0051] The six cameras at different angles need to cover a 360° viewing angle around the vehicle;

[0052] The internal and external parameter calibration uses Zhang Zhengyou calibration method;

[0053] The image data format adopts png (Portable Network Graphics).

[0054] S2: Extract multi-scale features from images at different angles;

[0055] Including the process of obtaining multi-scale features and fusion of spatiotemporal features;

[0056] See also Figure 2 In one embodiment, the specific process of multi-scale feature extraction is:

[0057] Firstly, a multi-scale feature acquisition model is established, which consists of a two-dimensional backbone network ResNet for basic feature extraction and a neck FPN for multi-scale variable objects;

[0058] In the model, for each input image of shape H×W×3, after passing through the multi-scale feature acquisition model, a Multi-scale features P2, P3, P4, P5.

[0059] In the above formula, H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image;

[0060] Then, a spatiotemporal feature fusion process is performed: the spatiotemporal feature fusion process is performed using a convolutional neural network;

[0061] The spatiotemporal fusion process is expressed as follows:

[0062] F=(P2,P3,P4,P5)

[0063] Use a 1×1 convolutional layer to process each spatial position and generate a weight feature map for each position at the current time step. Then, the weights are processed by the Sigmoid activation function to obtain the weight feature map

[0064] W t =Sigmoid(Conv(F))

[0065] For the previous time step W' t-1 The weighted feature maps are weighted summed to obtain weighted features with spatiotemporal information

[0066] W′ t =αW t +βW′ t-1

[0067] Multiply the generated weight feature map by the original feature map element by element, and we get

[0068] F weighted =F⊙W′ t

[0069] The sampled features are concatenated together and a 1×1 convolution layer is applied to obtain the final fused feature map

[0070] Fi =Conv(Concat(F weighted ))

[0071] Where Conv(·) represents the convolution operation, Sigmoid(·) represents the Sigmoid activation function, α and β are weight factors, ⊙ represents element-by-element multiplication, and Concat(·) represents the feature map cascade operation.

[0072] S3: Project multi-scale features into BEV space;

[0073] See also Figure 3 The specific process of generating the image feature point cloud and the BEV feature is as follows:

[0074] The image feature point cloud is generated as follows:

[0075] The image feature point cloud generation model is established. The coordinates of the generated view frustum point cloud in the image coordinate system and the corresponding depth values ​​are described by (u, v, d). The Efficientnet-B0 backbone network is used to extract features from the surround view image.

[0076] In the model,

[0077]

[0078] Among them, K represents the camera internal parameter, X cam , Y cam , Z cam is the 3D coordinate in the camera coordinate system, X ego , Y ego , Z ego is the 3D coordinate in the ego-vehicle coordinate system, (R, t) is the camera external parameter;

[0079] The BEV features are generated as follows:

[0080] The global pooling operation is used to summarize the global information of the entire feature map to obtain the overall background and context of the image. For the input feature F, we can get the global pooling feature:

[0081]

[0082] The local pooling operation extracts feature information from local regions of the input feature map to capture local details of the target:

[0083]

[0084] By adjusting the pooling area, traversing the entire feature map, we get pooling feature maps from different areas, and then connect these pooling feature maps together:

[0085] F local (c) = Conv{concat[f local (i,j,c)]}

[0086] Use Transformer to fuse the two types of features

[0087]

[0088] Where F(i,j,c) is the feature extracted by Efficientnet-B0, F global (c) is the feature obtained by global pooling, F local (c) is the feature obtained by local pooling, F BEV is the generated BEV feature, i and j represent the row and column indices of the feature map, H and W represent the height and width of the feature map, c represents the channel index, and k represents the pooling window size.

[0089] S4: Design a detection head to decouple the category and location information of objects in the input feature map;

[0090] See also Figure 4 , the detection head operation process is:

[0091] Specifically, L layers and set-based computation are used to generate bounding box estimates from 2D feature maps, each of which includes the following steps:

[0092] 1) Predict a set of bounding box centers associated with the object query.

[0093] 2) Project these centers into all feature maps using the camera transformation matrix.

[0094] 3) Sample features via bilinear interpolation and incorporate them into the object query.

[0095] 4) Use multi-head attention to describe the interaction of objects.

[0096] S5: Evaluate and optimize the model:

[0097] The evaluation and optimization include parameter tuning and iterative optimization. Specifically, the parameter tuning includes performing parameter tuning using a parameter tuning tool or algorithm;

[0098] The iterative optimization refers to the feedback of evaluation results and parameter tuning to further improve and optimize the model.

[0099] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] The present invention designs an enhanced small target detection network based on visual sensors. The method combines the small target detection method and the BEV perception algorithm, and utilizes multi-angle camera data and a neural network to enable the vehicle to obtain a perception of surrounding objects centered on itself, thereby providing data support for subsequent positioning, path planning, and vehicle control.

[0101] Secondly, the present invention designs a feature projection network, which rotates the features in the camera coordinate system to the vehicle coordinate system through a rotation matrix, and then performs global pooling and local pooling on the multi-scale features to construct small target features. The Transformer is used to summarize the overall background and context information of the global features and the small target information of the local features, and project them into the BEV space.

Claims

1. An environmental perception method for autonomous driving vehicles with enhanced small target detection, Features: The following steps are involved: S1: Obtain all-round images around the vehicle through multi-angle visual sensors; S2: Extract multi-scale features from images at different angles; S3: Project multi-scale features into BEV space; S4: Design a detection head to decouple the category and location information of objects in the input feature map; S5: Evaluate and optimize the model.

2. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 1, Features: In the S1, six cameras are used to obtain images at different angles centered on the autonomous driving vehicle, using png as the data format.

3. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 1, Features: The S2 includes the process of acquiring multi-scale features and fusing spatiotemporal features.

4. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 3, Features: The specific steps in S2 are as follows: Firstly, a multi-scale feature acquisition model is established, which consists of a two-dimensional backbone network ResNet for basic feature extraction and a neck FPN for multi-scale variable objects; In the model, for each input image of shape H×W×3, after passing through the multi-scale feature acquisition model, a The multi-scale features P2, P3, P4, P5 of the image; H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image; Then, a spatiotemporal feature fusion process is performed: the spatiotemporal feature fusion process is performed using a convolutional neural network; The spatiotemporal fusion process is expressed as follows: F=(P2,P3,P4,P5) W t =Sigmoid(Conv(F)) W′ t =αW t +βW′ t-1 F weighted =F⊙W’ t F i =Conv(Concat(F weighted )) Where Conv(·) represents the convolution operation, Sigmoid(·) represents the Sigmoid activation function, α and β are weight factors, ⊙ represents element-by-element multiplication, and Concat(·) represents the feature map cascade operation.

5. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 1, Features: The S3 includes image feature point cloud generation and BEV feature generation.

6. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 5, Features: The specific steps in S3 are as follows: Firstly, a model for generating image feature point cloud is established. The coordinates of the generated view cone point cloud in the image coordinate system and the corresponding depth values ​​are described by (u, v, d). The Efficientnet-B0 backbone network is used to extract features from the surround view image. In the model, Among them, K represents the camera internal parameter, X cam , Y cam , Z cam is the 3D coordinate in the camera coordinate system, X ego , Y ego , Z ego is the 3D coordinate in the ego-vehicle coordinate system, (R, t) is the camera external parameter; Then, BEV feature generation is performed: the feature generation process uses global pooling and local pooling, which is expressed as follows: F local (c)=Conv{oncat[f local (i,j,c)]} Where F(i,j,c) is the feature extracted by Efficientnet-B0, F gloval (c) is the feature obtained by global pooling, F local (c) is the feature obtained by local pooling, F BEV is the generated BEV feature, i and j represent the row and column indices of the feature map, H and W represent the height and width of the feature map, c represents the channel index, and k represents the pooling window size.

7. According to claim 1, an environment perception method for an autonomous driving vehicle with enhanced small target detection, Features: The S4 includes generating bounding box estimates from 2D feature maps using L layers and set-based computation.

8. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 7, Features: Each layer needs to predict a set of bounding box centers associated with the object query, project these centers into all feature maps using the camera transformation matrix, sample features through bilinear interpolation, merge them into the object query, and finally use multi-head attention to describe the interactions of objects to generate 3D bounding boxes.

9. According to claim 1, an environment perception method for an autonomous driving vehicle with enhanced small target detection, Features: The S5 includes parameter tuning and iterative optimization.

10. The method for environmental perception of an autonomous driving vehicle with enhanced small target detection according to claim 9, Features: The parameter tuning includes using a parameter tuning tool or algorithm to perform parameter tuning; The iterative optimization: further improve and optimize the model based on the evaluation results and feedback from parameter tuning.

Images