A transformer-based autonomous driving target detection method and system

Abstract

The present invention discloses a transformer-based target detection method for autonomous driving. This method overcomes the problems of high computational complexity, slow convergence, and redundant information in the pseudo-3D detection of vehicles in prior art transformer-based target detection methods. The method comprises the following steps: S1: using a transformer to predict 2D detection boxes and related vehicle information for different target vehicles; S2: obtaining the pseudo-3D box of the vehicle based on the prediction results. A transformer-based autonomous driving target detection system is also provided. A sparse attention mechanism is added when calculating self-attention at different granularities to reduce computational complexity. Furthermore, based on the 2D detection, three key points are predicted to reduce repeated predictions in the network.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a transformer-based automatic driving target detection method and system. Background Art

[0002] Object detection technology is an essential component of autonomous driving. CNN (convolutional neural network)-based object detection algorithms focus on spatial information and edge details, but are limited by local features. Consequently, numerous researchers have addressed issues such as spatial information extraction and lightweight design through structural design (e.g., FPN) and skip-bridging (e.g., ResNet). Its main structure can be divided into four parts: the input module, which provides the network input and includes image preprocessing; the backbone (neural network model) module, which extracts features from the input image and generates feature maps; the neck module (a key link in the object detection framework), which integrates feature maps at different levels; and the head module, which performs predictions on the feature maps. However, establishing global relationships remains a challenge.

[0003] Transformer-based models have shone brightly in natural speech processing and are currently achieving significant breakthroughs in image processing. The core component of the Transformer framework is the self-attention mechanism, which is used to build connections between objects and maintain a global perspective. Self-attention works by setting three trainable weight matrices, which are multiplied by the input vector to produce three vectors Q, K, and V. The output is then normalized and multiplied by V after taking the inner product of all pixels in Q and K. This ensures that each pixel contains global features. This matrix operation is parallelizable and more efficient than convolution. However, existing transformer-based object detection methods fail to fully exploit the advantages of transformers and are computationally intensive, resulting in slow convergence during training. For example, DETR (End-to-End Object Detection Based on Transformers) and its variants use a CNN backbone to perform encoding and decoding to predict object categories and bounding boxes. These methods do not integrate multi-scale image features, resulting in lower accuracy than using a transformer as the backbone. Directly using a transformer-based network, such as the SWIN transformer, as the backbone of the network is computationally intensive and increases training convergence time.

[0004] Autonomous driving technology requires more than just a 2D bounding box of the target vehicle. Sometimes, to obtain better three-dimensional information output, pseudo-3D information of the vehicle is also required. The pseudo-3D bounding box predicts the pseudo-3D vertices of the vehicle in 2D coordinates. Using the coordinate points of the pseudo-3D bounding box and the vehicle's true dimensions, the image coordinates are mapped to the real coordinate system to assist in vehicle target positioning. Existing technologies for pseudo-3D vehicle detection typically predict pseudo-3D bounding boxes based on 2D detection, which contains redundant information. This is because the 2D bounding box and pseudo-3D bounding box have overlapping vertices, meaning these vertices belong to both the 2D bounding box and the pseudo-3D bounding box, resulting in duplicate detection content and increasing the network's learning parameters. For example, on April 28, 2020, the China Patent Office disclosed an invention entitled "An autonomous driving target detection system and method based on deep learning and binocular camera", with the publication number CN111079586A. The invention uses a binocular camera to collect images and point cloud information, and transmits them to the image processing module via a data cable for real-time detection. The image processing module is a detection module constructed after using deep neural network training data. It processes the collected images to achieve positioning, recognition and classification. Finally, the classification results are spliced ​​into a video through the result output module and output, completing the detection of road conditions in autonomous driving. Summary of the Invention

[0005] The purpose of the present invention is to overcome the problems of large computational complexity, slow convergence, and redundant information in the vehicle pseudo-3D detection of transformer-based target detection methods in the prior art. A transformer-based autonomous driving target detection method and system are provided. A sparse attention mechanism is added when calculating self-attention at different granularities, which reduces the computational complexity. At the same time, on the basis of 2D detection, three key points are predicted to reduce the repeated predictions of the network.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a transformer-based autonomous driving target detection method, characterized in that it includes the following steps:

[0007] S1: Use transformer to predict the 2D detection box and related vehicle information of different target vehicles;

[0008] S2: Get the pseudo 3D box of the vehicle based on the prediction results.

[0009] The Transformer model is an existing technology that uses an attention mechanism to improve model training speed. This paper uses the Transformer to predict the 2D bounding box of a target vehicle in autonomous driving scenarios. Based on 2D detection, it predicts three key points. These points are then combined with the vertices of the 2D bounding box to establish an image coordinate system. This mathematically generates pseudo-3D vertices, reducing repeated predictions within the network.

[0010] As an example, in step S1, the predicted relevant information of different target vehicles specifically includes: the four vertices (e, g, n, m) of the vehicle's 2D detection box, the key point a (x a ,y a ), b(x b ,y b ) and the body dividing point c(x c ,y c Based on the 2D detection, three more key points are predicted: two tire key points and one body boundary point. Then, an image coordinate system is established with the 2D box vertices. Pseudo-3D vertices are obtained mathematically to reduce repeated predictions in the network.

[0011] Preferably, in step S1, transformer is used to predict the 2D detection boxes and related vehicle information of different target vehicles:

[0012] S1.1: Input a vehicle body image and use several Stage modules with sparse attention mechanisms to extract features from the input image to obtain an initial feature map. After each stage, the feature map size is halved and the dimension is doubled.

[0013] S1.2: The output features of different stage modules are fused to obtain feature information of different scales. Since the feature map of each stage has different effects on target detection, the outputs at different scales are weighted and summed with different weights to output feature maps of the same size.

[0014] S1.3: By changing the number of channels through 1×1 convolution, different tasks of the feature map are classified and recognized to obtain the 2D detection frames of different target vehicles, as well as the key points of the vehicle tires and the dividing points of the vehicle body.

[0015] Preferably, in step S1.1, several Stage modules with sparse attention mechanisms are used:

[0016] The input image is divided into n×n patches (generally referring to some sub-image blocks in the image), which are sequentially passed through four stage modules with sparse attention mechanism. Each stage module consists of a patch embedding layer and a Focal Sparse Transformer (sparse attention) module. Each stage module outputs an initial feature map; after each stage, the size of the feature map is halved and the dimension is doubled.

[0017] Preferably, in step S1.1, a sparse attention mechanism is added:

[0018] A1: Use different sizes of fine-grained features to pool the feature maps into corresponding matrices;

[0019] A2: Apply correlation constraints to the matrix: each element only pays attention to its relative radius K and K+2i (K is a hyperparameter, i = 0, 1, 2, etc.). That is, except for the relative radius not exceeding K and the relative radius of K+2i, the attention value is set to 0.

[0020] The Focal Sparse Transformer module is an improvement on the Facial Transformer. In the traditional Transformer model, the Focal Sparse Self-Attention module (FSSA) replaces the original focal self-attention component after the LayerNorm layer. The main difference lies in the addition of sparse attention to reduce the computational complexity when calculating local fine-grained self-attention and global coarse-grained attention. Because the correlation between coarse-grained windows is weaker, the K value should be larger than that of fine-grained windows.

[0021] Preferably, in step S1.2, feature maps of the same size are output:

[0022] S1.2.1: Each time the input image passes through a Stage module, the initial feature map is doubled using an upsampling block (such as linear interpolation or convolution) to obtain an intermediate feature map. Specifically: after passing through the first Stage module, no upsampling is required. After passing through the second Stage module, the initial feature map needs to be doubled. After passing through the third Stage module, the initial feature map needs to be quadrupled, and so on.

[0023] S1.2.2: Add the intermediate feature map to the initial feature map output by the previous Stage module to obtain the final output feature map of the previous Stage module. Specifically: as in the third Stage module, the initial feature map is doubled by the sampling module to obtain an intermediate feature map. The intermediate feature map is the same size as the initial feature map at the output end of the second Stage module. That is, the initial feature map of the second Stage module is added to the doubled intermediate feature map of the third Stage module to obtain the output feature map. The output feature map is then doubled by the upsampling block until it is the same size as the initial feature map at the output end of the first Stage module, thereby obtaining the final output feature map of the second Stage module.

[0024] S1.2.3: Add a learnable influence factor to the final output feature map of each stage module, multiply it with the output of the corresponding stage module, and sum it up to perform multi-scale feature fusion to obtain a feature map of the same size.

[0025] Preferably, in step S2, the tire key points and the vehicle body dividing points are both provided with category labels, which can be used to distinguish the left and right tires and the front and rear vehicle dividing points. These categories divide the vehicle driving status into eight categories: head left, head right, tail left, tail right, head only, tail only, left only, and right only. Finally, the detection results are post-processed to obtain a pseudo 3D frame.

[0026] Preferably, the step S2 is further expressed as:

[0027] S2.1: Set vertex coordinates based on the detected vertex information and establish an image coordinate system in the image based on the vertex coordinates, where the origin is the vertex in the upper left corner of the vehicle image. This is for the vehicle in the tail-left category, and does not necessarily mean that the origin must be the vertex in the upper left corner of the vehicle image.

[0028] S2.2: Calculate the equation of the line L(a,b) = F(x a ,y a ,x b ,y b ), calculate the straight line L(a,b) and the straight line y=y c The intersection point i of the straight line L(a,b) and the straight line x=x e The intersection point j;

[0029] S2.3: The projection of point c on the x-axis intersects the 2D box at point f. Point g, downward along the y-axis, intersects point i along the x-axis to the right at point h (x g ,y i ), get the vertices (e, f, g, h, i, j) of the pseudo 3D box;

[0030] S2.4: All vehicles are calculated according to the steps S2.1-S2.3.

[0031] Using two tire key points and a vehicle body dividing line point, an image coordinate system is established with the 2D box vertices, and pseudo 3D vertices are obtained mathematically to reduce repeated predictions of the network.

[0032] Preferably, in the step S2, if the tire points are not visible, the pseudo 3D frame overlaps with the 2D frame, and repeated detection is not required, thereby reducing repeated predictions of the network.

[0033] A transformer-based autonomous driving target detection system, comprising:

[0034] Feature extraction module: connected with the feature fusion module, used to extract global and local information features of the image;

[0035] The feature extraction module includes several stage modules, which include:

[0036] Sub-image block embedding layer: embeds a learnable position information parameter for each sub-image block and projects it into the dimensional channel space through a convolution with a convolution kernel and a stride of n. After each sub-image block embedding layer, the feature map size is halved and the dimension is doubled.

[0037] Sparse attention mechanism module: reduces the amount of computation when calculating local fine-grained self-attention and global coarse-grained attention. The sparse attention used in this invention can use other sparse attention methods;

[0038] Feature fusion module: connected to the detection module, used to fuse features of different scales;

[0039] Detection module: classifies and locates the input features, including box classification, box regression, point classification, and point regression. The detection module can perform classification and regression for different tasks as needed.

[0040] Therefore, the present invention has the following beneficial effects: 1. The present invention uses a transformer-based target detection algorithm, uses an improved focal transformer as a feature extractor of the network, and adds a sparse attention mechanism when calculating self-attention at different granularities, thereby reducing the additional computational effort brought by the focal transformer; 2. Learnable influencing factors are added to output feature maps of different scales, and different weights are used for weighted summation to facilitate capturing the features of targets of different sizes in the image; 3. On the basis of 2D detection, three key points are predicted, including two tire key points and one vehicle body dividing line point, and then an image coordinate system is established with the 2D box vertices, and pseudo 3D vertices are obtained mathematically, thereby reducing repeated predictions of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Detailed operation flow chart of the method of the present invention;

[0042] Figure 2 It is a structural block diagram of the system of the present invention;

[0043] Figure 3 This is the transformer network structure framework diagram of the present invention;

[0044] Figure 4 This is a schematic diagram of the pseudo 3D frame structure of the target vehicle of the present invention;

[0045] In the figure: 1. Feature extraction module; 2. Feature fusion module; 3. Detection module; 4. Stage module; 5. Sub-image block embedding layer; 6. Sparse attention mechanism module. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] like Figure 1 In the illustrated embodiment, a transformer-based autonomous driving target detection method can be seen, and its operation process is as follows: Step 1, using the transformer to predict the 2D detection box and related vehicle information of different target vehicles; Step 2, obtaining the pseudo 3D box of the vehicle based on the prediction results.

[0048] The transformer predicts the 2D frame of the target vehicle in the autonomous driving scenario. Based on the 2D detection, three key points are predicted. Then, an image coordinate system is established with the vertices of the 2D frame. Pseudo-3D vertices are obtained mathematically to reduce repeated predictions of the network.

[0049] like Figure 2 The illustrated embodiment shows a transformer-based autonomous driving object detection system. The entire system framework consists of three modules: a feature extraction module, a feature fusion module, and a detection module. The feature extraction module extracts global and local image features; the feature fusion module fuses features at different scales; and the detection module classifies and locates objects based on input features.

[0050] The following further illustrates the technical solutions and technical effects of the present invention through specific examples. The following examples are intended to explain the present invention, but the present invention is not limited to the following examples.

[0051] like Figure 3 As shown:

[0052] Step 1: Use transformer to predict the 2D detection box and related vehicle information of different target vehicles

[0053] First, the input image is divided into sub-image blocks of size 4×4, and then input into the feature extraction module, which includes 4 stage modules. Each stage module consists of a sub-image block embedding layer and a sparse attention module. The sub-image block embedding layer embeds a learnable position information parameter for each sub-image block and projects it into the dimensional channel space through a convolution kernel and a stride of 4. After each stage, the size of the feature map is halved and the dimension is doubled.

[0054] The sparse attention module is an improvement on the facial transformer. It replaces the original focal self-attention with the FASA block (Focal Sparse Self-Attention) after the original LN (LayerNorm) layer. The main difference is that the FASA block adds sparse attention to reduce computational overhead when calculating local fine-grained self-attention and global coarse-grained attention. Specifically, FASA first pools feature maps into corresponding matrices using different fine-grained sizes. However, not all tokens in each matrix have useful correlations, so correlation constraints are applied. Each element only calculates attention with elements within a relative radius of K and K+2i (K is a hyperparameter, i = 0, 1, 2, etc.). That is, the attention value is set to 0 except for positions with a relative radius of less than K and a relative radius of K+2i. Since the correlation between coarse-grained windows is weaker, the value of K should be larger than that of fine-grained windows.

[0055] The feature fusion module fuses the output features of different stage modules to obtain feature information of different scales. Since the feature maps of each stage have different effects on target detection, the outputs at different scales are weighted and summed with different weights. Specifically, in order to ensure that each stage finally outputs a feature map of the same size, an upsampling block (such as linear interpolation or deconvolution) is used to double the feature map each time; it is then added to the feature map of the same size in the previous layer, and the learnable influence factors α, β, γ, and δ are added to the final output of stage1, stage2, stage3, and stage4. These factors are multiplied with the output of the corresponding stage and summed to perform multi-scale feature fusion.

[0056] Specifically, in the third stage module, the initial feature map is doubled using the sampling module to obtain an intermediate feature map. This intermediate feature map has the same size as the initial feature map at the output of the second stage module. The initial feature map of the second stage module is added to the doubled intermediate feature map of the third stage module to obtain the output feature map. The output feature map is then doubled using the upsampling block until it has the same size as the initial feature map at the output of the first stage module. This yields the final output feature map of the second stage module. The learned influence factor β is then added, multiplied by the output of the corresponding stage, and summed to perform multi-scale feature fusion. Finally, in the detection module, the number of channels is changed through 1×1 convolution to perform classification and recognition for different tasks.

[0057] Step 2: Get the pseudo 3D frame of the vehicle based on the prediction results

[0058] This paper primarily focuses on 2D target detection and pseudo-3D vehicle detection in autonomous driving scenarios. Using the transformer network in the first step, the 2D detection boxes of different target vehicles, as well as the key points of their tires and body demarcation points, are predicted. Both the tires and body demarcation points are labeled with categories, allowing them to distinguish between left and right tires and front and rear vehicle demarcation points. These categories categorize vehicle driving states into eight categories: head left, head right, tail left, tail right, head only, tail only, left only, and right only. Finally, the detection results are post-processed to produce pseudo-3D boxes.

[0059] The specific processing results are as follows Figure 4 As shown, the vehicle belongs to the tail left category. Points (e, g, n, m) are the four vertices of the detected vehicle 2D box, point a (x a ,y a ), b(x b ,y b ) is the key point of the tire, point c(x c ,y c ) is the dividing point of the vehicle body. An image coordinate system is established in the image based on the coordinates of each vertex. The origin is the vertex in the upper left corner of the image. The x-axis is to the right of the origin, and the y-axis is downward from the origin. The linear equation L(a, b) = F(x a ,y a ,x b ,y b ), calculate the straight line L(a,b) and the straight line y=y c The intersection point i of the straight line L(a,b) and the straight line x=x e The intersection point j, the projection of point c on the x-axis intersects the 2D box at point f, and point g goes down along the y-axis and intersects point i to the right along the x-axis at point h (x g ,y i ), and obtain the vertices (e, f, g, h, i, j) of the pseudo 3D box. The other seven cases are calculated in the same way. If the tire points are not visible, the pseudo 3D box coincides with the 2D box.

[0060] This paper proposes a Transformer-based object detection method for autonomous driving. This method uses an improved FocalTransformer as the network's feature extractor. To reduce the extra computational overhead of the existing FocalTransformer, a sparse attention mechanism is added to the self-attention calculations at different granularities. To capture the features of objects of varying sizes in an image, learnable influencing factors are added to the output feature maps at different scales, and weighted summation is performed using different weights to improve the Transformer's performance in image object detection. Furthermore, the vertices of the vehicle's pseudo-3D bounding box are determined by establishing an image coordinate system using 2D detection results and vehicle key points, reducing repeated predictions in the network.

[0061] The embodiment described above is only a preferred solution of the present invention and does not limit the present invention in any form. Other variations and modifications are possible without exceeding the technical solution described in the claims.

Claims

1. A transformer-based autonomous driving target detection method, characterized by: It includes the following steps S1: Use transformer to predict the 2D detection box and related vehicle information of different target vehicles; S1 includes: S1.1, input vehicle body image, use several Stage modules with sparse attention mechanism to extract features from the input image, and obtain the initial feature map; S1.2, fuse the output features of different Stage modules to obtain feature information of different scales, perform weighted summation on the outputs of different scales with different weights, and output feature maps of the same size; outputting feature maps of the same size includes: S1.2.1, adding the initial feature map to the intermediate feature map of the same size output by the next Stage module to obtain the output feature map, and using the upsampling block to expand the output feature map to the same size as the initial feature map output in the first Stage module to obtain the final output feature map of the Stage module; specifically, every time the input image goes through an additional Stage module, the upsampling block is used to double the initial feature map to obtain the intermediate feature map; S1.2.2, add a learnable influence factor to the final output feature map of each Stage module, multiply and sum it with the output of the corresponding Stage module to perform multi-scale feature fusion to obtain a feature map of the same size; S2: Get the pseudo 3D box of the vehicle based on the prediction results.

2. The transformer-based autonomous driving target detection method according to claim 1, characterized in that: In step S1, the predicted information specifically includes: the four vertices (e, g, n, m) of the vehicle's 2D detection box, the key point a (x a ,y a ), b(x b ,y b ) and the body dividing point c(x c ,y c ).

3. The transformer-based autonomous driving target detection method according to claim 1, characterized in that: The specific steps of step S1 also include: S1.3: By changing the number of channels through 1×1 convolution, different tasks of the feature map are classified and recognized to obtain the 2D detection boxes of different target vehicles and related vehicle information.

4. The transformer-based autonomous driving target detection method according to claim 3, characterized in that: In the step S1.1: The input image is divided into n×n sub-image blocks, which are sequentially passed through four stage modules with sparse attention mechanism. Each stage module outputs an initial feature map. After each stage, the size of the feature map is halved and the dimension is doubled.

5. The transformer-based autonomous driving target detection method according to claim 3 or 4, characterized in that: In the step S1.1, the attention value is set to 0 except for the positions with a relative radius not exceeding K and a relative radius of K+2i.

6. The transformer-based autonomous driving target detection method according to claim 2, characterized in that: In step S2, the tire key points and the vehicle body dividing points are both provided with category labels, and the category labels divide the vehicle driving status into 8 categories: head left, head right, tail left, tail right, head only, tail only, left only, and right only.

7. The transformer-based autonomous driving target detection method according to claim 2, characterized in that: The step S2 is further expressed as: S2.1: Set vertex coordinates based on the detected vertex information, and establish an image coordinate system in the image based on the vertex coordinates, where the origin is the upper left vertex of the vehicle image, the x-axis direction is to the right of the origin, and the y-axis direction is downward from the origin; S2.2: Calculate the equation of the line L(a,b) = F(x a ,y a ,x b ,y b ), calculate the straight line L(a,b) and the straight line y=y c The intersection point i of the straight line L(a,b) and the straight line x=x e The intersection point j when S2.3: The projection of point c on the x-axis intersects the 2D box at point f. Point g, downward along the y-axis, intersects point i along the x-axis to the right at point h (x g ,y i ), get the vertices (e, f, g, h, i, j) of the pseudo 3D box; S2.4: All vehicles are calculated according to the steps S2.1-S2.

3.

8. The transformer-based autonomous driving target detection method according to claim 7, characterized in that: In the step S2, if the tire points of the vehicle are not visible, the pseudo 3D frame and the 2D frame are overlapped.

9. A transformer-based autonomous driving target detection system, applied to the transformer-based autonomous driving target detection method according to any one of claims 1 to 8, characterized in that: include: Feature extraction module: extracts global and local information features of the image; The feature extraction module includes several stage modules, which include: Sub-image block embedding layer: embeds a learnable position information parameter for each sub-image block and projects it into the dimensional channel space through a convolution with a convolution kernel and a stride of n; Sparse attention mechanism module: reduces the amount of computation when calculating local fine-grained self-attention and global coarse-grained attention; Feature fusion module: fuses features of different scales; Detection module: classifies and locates targets based on input features.

Images