A panoramic driving perception method based on deep learning

Abstract

The application belongs to the field of intelligent driving, and particularly relates to a panoramic driving perception method based on deep learning. The method comprises the following steps: after a driving image to be perceived is subjected to a series of feature extraction, a mixed attention mechanism module is used to perform mixed attention processing on multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image, so that mixed attention lane line semantic features and mixed attention drivable semantic features of the driving image are obtained; and the feature maps are respectively input into a target detection head and a semantic segmentation head module. After the output images of different tasks obtained are subjected to post-processing by a post-processing module, finally, the output images of the passing space target detection, lane line semantic segmentation and drivable area semantic segmentation after superposition are output. The mixed attention mechanism module proposed in the application can effectively alleviate the interference of the background on the semantic segmentation result, and can better restore the boundary geometric information between the lane line and the drivable area.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving, specifically relating to a panoramic driving perception method based on deep learning. Background Technology

[0002] With the rapid development of artificial intelligence in the automotive industry, autonomous driving technology has become a hot topic in both academia and industry, attracting widespread attention both domestically and internationally. To advance autonomous driving technology from science fiction to reality and seize the commanding heights in this field, various countries have implemented corresponding strategic plans. my country, in its "Made in China 2025" plan, has explicitly identified intelligent connected vehicles as a key development direction and has issued road test permits for some autonomous vehicles. Furthermore, some domestic and international companies have successfully applied autonomous driving technology in specific scenarios, such as DeepDrive's intelligent taxis and Baidu's intelligent trucks. In recent years, autonomous driving technology has achieved milestone progress, but since its inception, controversy surrounding it has never ceased. Even so, this future technology continues to develop rapidly amidst skepticism and challenges, becoming one of the biggest technological revolutions of our time, profoundly changing people's travel habits and social structures.

[0003] Road scene understanding, one of the most challenging tasks in autonomous driving technology, is a prerequisite for vehicles to achieve autonomous driving. Currently, road scene understanding technology is mainly based on two types of sensors: LiDAR and onboard cameras. While LiDAR has high detection accuracy, its high cost hinders its widespread adoption. In contrast, onboard cameras, which are low-cost and resemble human visual perception, are widely used in the field of intelligent vehicles. By collecting information about the surrounding traffic scene using onboard cameras, lane features are extracted, and each lane line, each drivable area, and obstacle is quickly and accurately located, guiding the vehicle to drive in the correct area, effectively avoiding risks, and reducing traffic accidents.

[0004] Currently, deep learning-based object detection algorithms and road scene recognition have become mainstream technologies in the autonomous driving stage. These algorithms primarily capture hidden scene semantic information by predicting background, foreground, lane lines, road and obstacle categories pixel by pixel. Therefore, they are more suitable for solving tasks such as object detection in traffic spaces, lane line recognition, and road recognition in complex and ever-changing real-world environments. Consequently, the core technologies of panoramic driving perception systems—namely, algorithms for traffic space object detection, lane line recognition, and drivable area recognition for road scene understanding—have migrated from traditional image processing techniques used in the assisted driving stage to deep learning-based object detection algorithms and road scene recognition. Improvements have been made to existing algorithms for traffic space object detection, lane line recognition, and drivable area recognition to address their limitations, such as limited detection scenarios, susceptibility to background interference, and low detection efficiency.

[0005] While deep learning-based object detection and road scene understanding algorithms have made significant progress, they still face two main challenges: the complex and ever-changing driving environment and inherent limitations of the algorithms themselves. First, complex environmental factors lead to poor data quality: 1) the influence of varying light intensity; 2) adverse weather conditions such as rain, snow, and fog; 3) the impact of complex road conditions, such as excessive road curvature and lane markings obscured by surrounding vehicles and shadows. Second, existing algorithms still face issues with accuracy, real-time performance, and reliability: 1) low object detection accuracy under complex environmental conditions; 2) unreasonable algorithm architecture design leading to wasted computational resources in feature extraction, making it difficult to guarantee real-time inference requirements; 3) generally high false detection rates, hindering practical application in autonomous driving.

[0006] In conclusion, the development of intelligent driving technology is timely; however, accurately performing tasks such as target detection in the traffic space, lane line recognition, and drivable area identification is crucial for driving safety. The emergence of deep learning technology has spurred rapid development in many traditional computer vision fields, and intelligent road detection systems can provide a safer road traffic environment. However, challenges such as inaccurate detection and real-time performance still exist. Summary of the Invention

[0007] This application proposes a deep learning-based panoramic driving perception method, which focuses on solving the following technical problems by designing a core algorithm for a panoramic driving perception system based on multi-scale convolution and attention mechanisms:

[0008] (1) The detection scenarios are limited and cannot be applied to the needs of autonomous driving. Currently, the mainstream publicly available lane line recognition datasets are only for simple assisted driving scenarios. However, one of the core technologies of the panoramic driving perception system is to use a deep learning model to solve three tasks at the same time: target detection in the traffic space, lane line recognition, and drivable area recognition. Therefore, it is necessary to have the original images and labeled data for all three tasks. In addition, a dataset that meets this condition not only needs to have labeling schemes for intersections and complex road conditions to adapt to autonomous driving scenarios that require automatic lane changing, entering and exiting intersections, but also needs to include more complex driving environments to adapt to driving scenarios in different regions.

[0009] (2) Current semantic segmentation-based lane line recognition and drivable area recognition algorithms are easily affected by background interference. When there is a large data imbalance between negative samples and positive lane line samples, semantic segmentation algorithms have difficulty recognizing small targets. At the same time, a large difference between the loss function's scale and the task can lead to poor semantic segmentation performance and increase the probability of missed detections and false detections. The hybrid attention mechanism module proposed in this invention can effectively alleviate the interference of background on semantic segmentation results.

[0010] (3) The detection efficiency of the recognition algorithm is low and cannot meet the real-time requirements; it lacks more supervisory information to train the network, such as the geometric information of lane lines and drivable areas. For most existing traffic space target detection, lane line recognition, and drivable area recognition algorithms, although they all have good detection results, each algorithm is independent of the others. When the three algorithms are executed simultaneously, not only is a greater computational cost required, but they also cannot utilize the information obtained by each other. The hybrid attention mechanism module proposed in this invention allows lane line semantic features and drivable semantic features to communicate with each other when performing semantic segmentation on a certain image at the same time, so as to better restore the boundary geometric information between lane lines and drivable areas.

[0011] The present invention provides a deep learning-based panoramic driving perception method, the method comprising:

[0012] Acquire the driving image to be perceived and preprocess the driving image;

[0013] The driving image is subjected to multidimensional feature extraction using a feature extraction subnetwork to obtain a multidimensional feature map of the driving image;

[0014] The multidimensional feature map of the driving image is fused using a feature fusion subnetwork to obtain the fused feature map of the driving image;

[0015] The multi-scale convolution module is used to extract features from the fused feature map of the driving image to obtain multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image.

[0016] The multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image are processed by a hybrid attention mechanism module to obtain hybrid attention lane line semantic features and hybrid attention drivable semantic features of the driving image.

[0017] The lane line semantic segmentation module is used to process the hybrid attention lane line semantic features of the driving image to obtain the lane line semantic segmentation result of the driving image.

[0018] The drivable region semantic segmentation module is used to perform feature processing on the hybrid attention drivable semantic features of the driving image to obtain the drivable semantic segmentation result of the driving image;

[0019] The spatial target detection module is used to detect and process the multi-dimensional features of driving images to obtain the spatial target detection results of driving images;

[0020] The spatial target detection results, lane line semantic segmentation results, and drivable semantic segmentation results are superimposed using the classification module to obtain the prediction result of the driving image to be perceived.

[0021] The beneficial effects of this invention are:

[0022] 1. The detection scenarios of this invention are diverse and can be applied to the needs of autonomous driving. It can simultaneously solve three tasks: target detection in the traffic space, lane line recognition, and drivable area recognition, and can adapt to a wider range of complex driving environments to suit driving scenarios in different regions.

[0023] 2. The hybrid attention mechanism module proposed in this invention can effectively alleviate the interference of background on semantic segmentation results.

[0024] 3. The recognition algorithm of this invention has high detection efficiency and can meet the real-time requirements. When the hybrid attention mechanism module proposed in this invention performs semantic segmentation on a certain image at the same time, the multi-scale lane line semantic features and the multi-scale drivable semantic features can communicate with each other, so as to better restore the boundary geometric information between the lane lines and the drivable area. Attached Figure Description

[0025] Figure 1 This is a structural diagram of the deep learning-based panoramic driving perception framework according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of a deep learning-based panoramic driving perception method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the pooled pyramid module structure used in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the hybrid attention module structure according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the cross-attention module structure according to a preferred embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Figure 1 This is a structural diagram of the deep learning-based panoramic driving perception framework according to an embodiment of the present invention, as shown below. Figure 1 As shown, in this embodiment of the invention, it mainly includes a feature extraction subnetwork, a feature fusion subnetwork, a multi-scale convolution module, a hybrid attention mechanism module, a lane line semantic segmentation module, a drivable area semantic segmentation module, a traffic space target detection module, and a classification module. These subnetworks and modules together constitute a deep learning-based panoramic driving perception network. By training and optimizing this perception network, the perception processing of the driving image to be perceived can be realized.

[0032] Based on this, embodiments of this application propose a deep learning-based panoramic driving perception method, which will be described in detail below.

[0033] Please see Figure 2 , Figure 2 This is a flowchart illustrating a panoramic driving perception method in an exemplary embodiment of this application.

[0034] In an exemplary embodiment, the detection method includes at least steps S110 to S190, which are described in detail below:

[0035] S110. Acquire the driving image to be perceived and preprocess the driving image;

[0036] In the embodiments of this application, when performing panoramic driving perception, image preprocessing is typically performed on each frame of the video captured by the vehicle-mounted camera to obtain the input driving image.

[0037] The preprocessing method of the present invention is used to eliminate irrelevant information in the original image, restore useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

[0038] S120. Multidimensional feature extraction is performed on the driving image using a feature extraction subnetwork to obtain a multidimensional feature map of the driving image;

[0039] In this embodiment of the application, the feature extraction subnetwork serves as the basic feature extraction module, namely... Figure 1 In Part ①, CSPDarknet is selected as the backbone network. The input image undergoes multiple convolutions and progressive downsampling by the basic feature extraction module to obtain feature maps FV1, FV2, and FV3 of the lane frontal view angle, which are then used as the input feature information for the multi-scale convolutional attention module.

[0040] S130. The multidimensional feature map of the driving image is fused using a feature fusion sub-network to obtain the fused feature map of the driving image;

[0041] In this embodiment of the invention, a feature fusion subnetwork is utilized. Figure 1 In Part ②, the feature fusion subnetwork can use common fusion modules, such as Faster RCNN, Mask RCNN, Yolov3, RetinaNet, Cascade RCNN, etc. In this embodiment, Faster RCNN can be used for feature fusion.

[0042] S140. Use a multi-scale convolution module to extract features from the fused feature map of the driving image to obtain multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image.

[0043] In this embodiment of the invention, multi-scale convolution is an operation in convolutional neural networks, namely... Figure 1 Part ③ of the diagram is used to extract features at different scales. This can be achieved by using convolutional kernels of different sizes in different convolutional layers. These kernels of different sizes can capture features from the input data at different scales. Multi-scale convolution can also be implemented using pooling layers, where pooling kernels of different sizes are applied to the input data to obtain feature representations at different scales. Multi-scale convolution is widely used in applications such as image classification, object detection, and semantic segmentation.

[0044] Since the targets to be detected, lane lines, and drivable areas in the traffic space may span a wider area or be subject to occlusion, hybrid attention mechanisms can improve recognition accuracy. However, traditional methods that only use preset convolutional kernel sizes as basic computational units are not well-suited for complex multi-target detection and recognition tasks.

[0045] However, for multi-scale convolutional modules, multiple convolutional kernels of different granularities or sizes will be used to learn deep-level feature information of the input state. For example... Figure 1 As shown in Part ③, various convolution kernels of different sizes can be created to accommodate different granularities, such as 1×2, 2×1, 2×2, m×n, etc.

[0046] In addition, dilated convolutional kernels with different magnifications can be used. Traditional convolution can only perceive input features within a local range, while dilated convolution can perceive features over a larger range and can increase the size of the receptive field without increasing computational cost. This allows the network to better understand the contextual information in the input, resulting in better results.

[0047] Therefore, in the preferred embodiment of this application, the Atrous Spatial Pyramid Pooling (ASPP) module is used as the multi-scale convolution module. The ASPP multi-scale convolution module is derived from the DeeplabV3+ model, which is an improved version of the DeeplabV3 model. This model is based on spatial pyramid pooling technology and, by integrating the advantages of multiple models, constructs a core network architecture to obtain a high-performance encoder and decoder based on a deep network structure.

[0048] Pooling pyramid module, such as Figure 3 As shown, it essentially consists of a 1×1 convolution (leftmost layer) + a pooling pyramid (three convolutions in the middle) + ASPP Pooling (three layers on the rightmost side). The dilation factor of each layer of the pooling pyramid can be customized, thus enabling flexible multi-scale feature extraction.

[0049] S150. The multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image are processed by the hybrid attention mechanism module to obtain the hybrid attention lane line semantic features and hybrid attention drivable semantic features of the driving image.

[0050] In this embodiment of the invention, the hybrid attention mechanism employed is partly inspired by the traditional self-attention mechanism. Self-attention is a widely used technique in deep learning and is one of the most important components of the Transformer model. The Transformer is a deep learning model for natural language processing that uses self-attention to capture correlation information in the input sequence, allowing the model to automatically assign weights to different positions when processing sequence data. Its formula is expressed as follows:

[0051] Q = W q *q (1)

[0052] K = Wk *k (2)

[0053] V = W v *v (3)

[0054]

[0055] In the formula, q, k, and v correspond to query, key, and value in the self-attention mechanism, respectively. In this patent, q, k, and v are the same input information. By calculating self-attention, this model can focus on the relationships between different parts of the input.

[0056] Axial attention is a self-attention mechanism applicable to computer vision, similar to the self-attention mechanism in the Transformer model. Axial attention not only offers more efficient computation but also amplifies the advantage of a large receptive field in attention models. Its core idea is to decompose two-dimensional attention into two one-dimensional attention along the height and width axes. Furthermore, axial attention adds relative position encoding to the self-attention model; these encodings are learnable. Experimental results show that attention-based image segmentation models combining axial attention and relative position encoding have the ability to encode the spatial structure of images. The attention computation method proposed by the axial attention model significantly reduces computational costs in visual tasks and enables the establishment of a fully self-attention model. However, this constraint sacrifices global connectivity, ensuring that the receptive field of the attention is no larger than that of a depthwise convolution with the same kernel size. Therefore, multiple axial attention mechanisms are employed to enable the network to perceive more feature information.

[0057] The axial attention mechanism employed in this patent ensures global connectivity and efficient computation. First, the axial attention layer along the image's width axis is simply described as a one-dimensional, position-sensitive self-attention mechanism, with a similar definition used for the height axis. Specifically, the axial attention layer along the width axis is defined as follows:

[0058]

[0059] The formula in equation (5) follows the axial attention mechanism, r q r k r v ∈R W×W Equation (5) describes the axial attention mechanism applied along the width axis of the tensor. A similar formula is used to apply the axial attention mechanism along the height axis. Both axial attention layers employ a multi-head attention mechanism. Together, they form a computationally efficient self-attention model.

[0060] However, for Transformers, the self-attention mechanism also results in a very large number of model parameters, requiring a large amount of data for training. If the amount of training data is insufficient, the model may overfit, leading to poor performance on the test set, and the same applies to the axial attention mechanism. For example, when training a model on a large dataset, it is easier to obtain query vectors, key vectors, and value vectors, especially for relative position encoding, where a larger dataset will make the learnable relative position encoding more accurate. However, for a network that can simultaneously train traffic space object detection, lane line semantic segmentation, and drivable area semantic segmentation, the amount of dataset is limited. Therefore, a controllable relative position encoding can be used, i.e., the modified equation (5) is as follows:

[0061]

[0062] In equation (6), G Q G K G V1 G V2 ∈R are learnable parameters that, together, create a gated axial attention mechanism to control the influence of relative positional encodings. That is, they are assigned higher weights to more accurately learned positional encodings.

[0063] One of the core aspects of the Transformer proposed in this patent is the hybrid attention mechanism. In this mechanism, the Transformer treats the input as a one-dimensional sequence, focusing solely on modeling the global context at all stages. This leads to a lack of detailed localization information in low-resolution features, resulting in coarse segmentation results. However, on the other hand, while convolutional architectures lack the ability to model long-range dependencies in images (even though using continuous pooling layers can increase the receptive field, it incurs significant structural loss), they provide a method for extracting low-level visual cues, which can effectively compensate for these spatial details.

[0064] Therefore, this patent proposes to combine multi-scale convolution with a hybrid attention mechanism. Specifically, a portion of the hybrid attention mechanism designed in this patent employs the aforementioned axial attention mechanism. Leveraging the powerful feature extraction capabilities of the axial attention mechanism in semantic segmentation tasks, and combining it with the flexible multi-scale convolution design and the dual advantages of convolutional neural networks, this significantly enhances the scaling invariance of images and the aggregation ability of local feature information. Figure 1(See Parts ③ and ④). Therefore, compared to traditional attention mechanisms that only use fixed convolution kernels for feature extraction, multi-scale convolution combined with attention mechanisms has a more comprehensive and flexible feature learning capability, which can enhance the correlation between lane lines and drivable area categories, thereby strengthening the expressive power of features. Features without the attention module and features processed by the multi-scale convolution module and attention module are fused to improve the robustness of the network; finally, through classification layers and upsampling, the predicted outputs of traffic space target detection, lane lines, and drivable area predictions are obtained. This not only allows for more precise classification of each pixel, effectively detecting the actual shapes of lane lines and drivable areas in the traffic space, but also greatly improves the accuracy of semantic segmentation and reduces the difficulty of post-processing.

[0065] Furthermore, this patent designs a Dual Transformer module based on a cross-attention mechanism ( Figure 1 Part ④ of the algorithm is used to fuse the feature information of lane lines and drivable areas, resulting in multiple feature maps that integrate various information, thus outputting more accurate prediction results. The overall structure of the axial attention mechanism combined with the cross-attention mechanism is as follows: Figure 4 As shown, these together constitute the hybrid attention mechanism proposed in this patent. The hybrid attention mechanism module performs hybrid attention processing on the multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image, including:

[0066] The axial attention mechanism is used to process the feature vectors of multi-scale lane line semantic features in a wide dimension and the feature vectors of multi-scale drivable semantic features in a wide dimension.

[0067] The axial attention mechanism is used to process the high-dimensional feature vectors of multi-scale lane line semantic features and the high-dimensional feature vectors of multi-scale drivable semantic features.

[0068] The cross-attention mechanism is used to process the feature vector of multi-scale lane line semantic features after axial attention in a wide dimension and the feature vector of multi-scale lane line semantic features after axial attention in a high dimension.

[0069] The cross-attention mechanism is used to process the feature vectors of multi-scale drivable semantic features after axial attention in a wide dimension and the feature vectors of multi-scale drivable semantic features after axial attention in a high dimension.

[0070] The cross-attention mechanism is used to process the feature vector of multi-scale lane line semantic features after the first cross-attention and the feature vector of multi-scale drivable semantic features after the first cross-attention.

[0071] The cross-attention mechanism is used to process the high-dimensional feature vector of the multi-scale lane line semantic features after the first cross-attention and the wide-dimensional feature vector of the multi-scale drivable semantic features after the first cross-attention.

[0072] Similar to self-attention, which is a crucial component of the Transformer, cross-attention is an essential part of the Dual Transformer module. Cross-attention allows learnable query vectors to interact with different image features. Specifically, the lane line semantic segmentation and drivable region semantic segmentation tasks can mutually acquire information from each other, enabling each resulting query vector to more clearly perceive the pixel information of the entire image.

[0073] The Dual Transformer module is primarily used to determine high-level semantic and detailed information between lane lines and drivable areas. For example, it relies on image context to determine lane line positions in scenarios where lane lines are obscured by vehicles or worn and discontinuous. The lane line semantic segmentation task also provides boundary information for the drivable area semantic segmentation task, enabling the semantic segmentation results to reconstruct the most realistic boundary divisions. The cross-attention mechanism is a crucial component of the Dual Transformer, and based on it, it further recovers detailed positional information of lane lines and drivable areas from high-resolution feature maps.

[0074] Therefore, as an important component of the Dual Transformer module, the cross-attention mechanism needs to pay attention to how the query, key, and value vectors of two different feature maps should interact with each other to achieve the best results when attention is applied between them in the height and width dimensions.

[0075] Figure 5This paper presents a detailed structure of the cross-attention mechanism in this embodiment. The implementation idea is as follows: First, obtain the wide-dimensional key and value vectors of the lane line semantic segmentation feature map, and then combine them with the wide-dimensional key and value vectors of the drivable area semantic segmentation feature map. The query vector obtained from the drivable area semantic segmentation feature map is then interacted with the key and value vectors after the combination of the two according to Equation (4) to obtain the feature map of the drivable area semantic segmentation feature map after the wide-dimensional attention (part 5 "Driving area feature map in widthdimension of cross-attention" in the figure); then, combine the high-dimensional key and value vectors of the new feature map with the high-dimensional key and value vectors of the lane line semantic segmentation feature map, and the query vector obtained from the new feature map is then interacted with the key and value vectors after the combination of the two according to Equation (4) to obtain the feature map of the new feature map after the high-dimensional attention (). Figure 5 In the section on "Driving area feature map in height dimension of cross-attention," the feature map undergoes a complete feature extraction process using a wide and high-dimensional attention mechanism. Then, following the same method, a new feature map for lane line semantic segmentation undergoes another complete wide and high-dimensional attention mechanism. This constitutes a complete feature extraction process using a cross-attention mechanism. In specific situations, this module can be stacked multiple times as needed to achieve better feature extraction results.

[0076] It is understandable that the stacking here can refer to using the feature vectors processed by the axial attention mechanism in each axial dimension as the input feature vectors before the cross-attention mechanism, or / and using the feature vectors processed by the cross-attention mechanism in the cross-dimensional dimension as the input feature vectors before the axial attention mechanism. This repeated stacking method achieves better feature extraction results, fully utilizing the advantages of the combined axial and cross-attention mechanisms, allowing communication between multi-scale lane line semantic features and multi-scale drivable semantic features. This enables a better reconstruction of the boundary geometry information between lane lines and drivable areas.

[0077] In some embodiments, the cross-attention processing of the features of the driving image after axial attention using the cross-attention mechanism module includes:

[0078] Obtain the key and value vectors of the lane line semantic feature map in the wide and high dimensions, and the key and value vectors of the drivable area semantic feature map in the wide dimension.

[0079] The concatenating connection layer of the cross attention unit is used to connect the key and value vectors of the lane line semantic feature map in the wide dimension, and the key and value vectors of the drivable area semantic feature map in the wide dimension, to obtain the key and value vectors of the drivable area semantic feature map after wide-dimensional attention.

[0080] The query vector of the drivable area semantic feature map is interacted with the key vector and value vector after wide-dimensional attention to obtain the feature map of the drivable area semantic feature map after wide-dimensional attention.

[0081] The concatenating connection layer of the cross attention unit is used to connect the key and value vectors of the lane line semantic feature map in high dimension, and the key and value vectors of the drivable area semantic feature map after wide-dimensional attention in high dimension, to obtain the key and value vectors of the drivable area semantic feature map after high-dimensional attention.

[0082] The query vector of the semantic feature map of the drivable area after wide-dimensional attention is interacted with the key vector and value vector after wide-dimensional attention to obtain the feature map of the semantic feature map of the drivable area after high-dimensional attention.

[0083] The concatenating connection layer of the cross-attention unit is used to connect the key and value vectors of the drivable area semantic feature map after high-dimensional attention to the key and value vectors of the lane line semantic feature map in the wide dimension, so as to obtain the key and value vectors of the lane line semantic feature map after wide-dimensional attention.

[0084] The query vector of the drivable area semantic feature map after high-dimensional attention is interacted with the key vector and value vector of the lane line semantic feature map after wide-dimensional attention to obtain the lane line semantic feature map after wide-dimensional attention.

[0085] The concatenating connection layer of the cross-attention unit is used to connect the key and value vectors of the drivable area semantic feature map after high-dimensional attention to the lane line semantic feature map after high-dimensional attention, so as to obtain the key and value vectors of the lane line semantic feature map after high-dimensional attention.

[0086] The query vector of the lane line semantic feature map after high-dimensional attention is interacted with the key vector and value vector of the lane line semantic feature map after high-dimensional attention to obtain the feature map of the lane line semantic feature map after high-dimensional attention.

[0087] Based on this, the specific formula for the cross-attention mechanism of this invention can be given next. The feature map of the image after passing through the basic feature extraction module is then passed through two feature extraction modules of the network, each composed of axial attention (one module is responsible for lane line semantic segmentation, and the other for drivable area semantic segmentation), resulting in two sets of feature vectors. The query values ​​of these two sets of feature vectors are then extracted. 1 ,key 1 ,value 1 and query 2 ,key 2 ,value 2 The query vector of another module is used as the query vector for cross-attention, and the corresponding key vector and value vector are combined together as the key vector and value vector of the cross-attention mechanism (as shown in Equation 8-9).

[0088] At this point, the Dual Transformer module obtains feature information from both the lane line semantic segmentation task and the drivable area semantic segmentation task. For example, for the lane line semantic segmentation task, it obtains feature information from the drivable area semantic segmentation task. The lane line semantic segmentation task can perceive more pixel information in the entire image, thereby improving its semantic segmentation accuracy. Next, the lane line semantic segmentation task, after passing through the cross-attention mechanism, and the drivable area semantic segmentation task, after passing through the axial attention mechanism, again pass through the cross-attention mechanism to achieve mutual communication between the lane line semantic segmentation task and the drivable area semantic segmentation task.

[0089] q = query 2 (7)

[0090] Mixed-k = concate(key) 1 ,key 2 (8)

[0091] Mixed-v = concate(value) 1 ,value 2 (9)

[0092] cross-att(q,Mixed-k,Mixed-v)=softmax(q(Mixed-k) T (Mixed-v)) (10)

[0093]

[0094] Equation (11) is a combination of equations (10) and (6). The core formulas involved in the multi-head attention mechanism in the Dual Transformer module are as follows:

[0095] self-att:Q i (0) =MultiHead(Q′) i-1 )

[0096] where Q′ i-1 =Attention(Q,K,V) (12)

[0097] FFN:Q i (1) =FFN(Q i (0) (13)

[0098] Cross-Attention:Q i (3) =MultiHead(Q i (2) )

[0099] where Q i (2) =cross-att(q,Mixed-k,Mixed-v) (14)

[0100] FFN:Q i (4) =FFN(Q i (3) (15)

[0101] In the formula, Q′ i-1 Q i (2) This represents the vector obtained by adding positional encoding and then encoding through a multi-head attention mechanism. Q i (0) Q i (3)These are two intermediate variables. FFN(x) in the above formula is the same as the definition of the standard Transformer decoder.

[0102] The Feed-Forward Network (FFN) module is a crucial component of the Transformer model. Following the axial attention mechanism at each layer, it introduces a non-linear transformation by mapping and combining features through two fully connected layers. A ReLU activation function separates these two layers, with the first fully connected layer having a larger output dimension and the second a smaller one. The FFN module enhances the model's expressive power by performing a non-linear transformation on the features at each location, allowing the model to better capture the semantics and features of the input. In this patent, the outputs of the axial and cross-attention mechanisms serve as the input to the FFN module. It can be viewed as a convolution-like operation that maps the feature maps used for semantic segmentation and object detection into a different feature space to improve representational power.

[0103] Axial attention mechanisms aim to focus more on capturing the intrinsic correlations within the data and reduce the influence of external information, making them an effective technique for improving model performance. In this patent, the axial attention mechanism focuses on its own task; the axial attention mechanism for lane line semantic segmentation focuses only on the lane line semantic segmentation module, and the axial attention mechanism for drivable area semantic segmentation focuses only on the drivable area semantic segmentation module. A cross-attention mechanism follows the axial attention mechanisms of both, achieving a synergistic effect. First, image information is encoded; then, the model performs axial attention on the image to capture the interactions between pixels; finally, the cross-attention mechanism allows each to utilize the information acquired by the other.

[0104] S160. The lane line semantic segmentation module is used to perform feature processing on the hybrid attention lane line semantic features of the driving image to obtain the lane line semantic segmentation result of the driving image.

[0105] In this embodiment of the invention, the lane line semantic segmentation result is determined by the lane line semantic segmentation head ( Figure 1 (Part ⑤) Obtain pixel-level spatial location information of lane lines, calculate the midpoint of lane lines on the same plane, and perform curve fitting in the vertical direction. Determine the thickness of the fitted curve by calculating the width of the lane lines in the horizontal direction. Figure 1(Part ⑦). Considering the occlusion problem of lane lines, the lane line fitting curve is dynamically extended to a uniform length in the vertical direction, based on the length of the longest detected lane line. The extension length parameter can be set by the user. Based on the prior knowledge that lane lines are parallel to each other, multiple pairs of horizontal spacing corrections are applied to the output lane lines. Finally, the fitted lane lines are plotted on the image to complete the lane line segmentation.

[0106] The calculation of the midpoints of lane lines on the same plane and the vertical curve fitting process includes traversing the midpoints of the lane lines, assigning the midpoints of lane lines that conform to the current vertical height range to the same baseline according to the traversal order, and assigning the midpoints of lane lines that do not conform to the current vertical height range to the next baseline, until all lane lines have been traversed; taking the midpoint of the leftmost lane line on the same plane as the reference point, taking the baseline containing the reference point as the baseline and forming a lane line fitting segment, sorting the baseline and other baselines according to their vertical height to determine the vertical positional relationship of each baseline; traversing the baselines other than the baseline and searching backwards, splicing the midpoints of adjacent lane lines belonging to the same baseline to form a lane line fitting segment. The process involves fitting lane lines to segments; using a reference point as the root node, the midpoints of the remaining lane lines on the reference line as the first child node of the root node, and the midpoints of the lane lines on other baselines as the second child node of the root node; identifying the relative positions between the midpoints of each lane line according to the positional relationships of the root node, each first child node, and the second child node; determining the relative positional relationships between the lane line fitting segments of the reference line and the lane line fitting segments of the other baselines according to the vertical positional relationships between the reference line and the other baselines; and determining the pixel-level semantic tree of the lane lines according to the relative positions between the midpoints of each lane line; and processing the semantic tree to obtain the semantic fitting result of the lane lines in the vertical direction.

[0107] In this embodiment of the invention, a baseline recognition method enables more accurate acquisition of lane line fitting segments. By utilizing the relative positional relationship between the main baseline and other baselines of the lane line fitting segments, a pixel-level semantic tree for the lane lines is determined. This semantic tree approach reduces the number of recognition and discrimination operations for lane line pixels, lowering complexity and improving the accuracy and efficiency of lane line fitting.

[0108] S170. The drivable region semantic segmentation module is used to perform feature processing on the hybrid attention drivable semantic features of the driving image to obtain the drivable semantic segmentation result of the driving image.

[0109] In this embodiment of the invention, for semantic segmentation of drivable areas, no post-processing is required on the results; the semantic segmentation results of drivable areas can be directly drawn onto the image.

[0110] S180. The multi-dimensional features of the driving image are detected and processed using the traffic space target detection module to obtain the spatial target detection result of the driving image.

[0111] In this embodiment of the invention, the target detection result in the passage space is determined by three target detection heads in the passage space ( Figure 1 (Part 6) Joint detection. Non-Maximum Suppression (NMS) is a technique for solving the overlapping bounding box problem, which will filter the detection results of the three target detection heads. Figure 1 (See Part 7). In passable space object detection tasks, an object may be surrounded by multiple candidate boxes, which often have a large overlap. The task of object detection is to find the most suitable box to represent the object. Without any processing, many duplicate or over-detection results will occur.

[0112] In some embodiments of the present invention, the present invention employs a non-maximum suppression method for processing. That is, before using the traffic space target detection module to detect and process the multi-dimensional features of the driving image and obtain the spatial target detection result of the driving image, the method includes: sorting the target detection results in descending order according to confidence; taking the detection box with the highest confidence and using it as the first element of the retained result list; comparing the remaining detection boxes with the elements in the retained result list in turn, and removing the overlapping region from the list if the IoU is greater than a set threshold; continuing to select detection boxes with high confidence from the remaining detection boxes and repeating the above process; increasing the penalty for confidence at the end of the detection box traversal until the final retained result is the detection box output by non-maximum suppression.

[0113] In a preferred embodiment of this invention, this patent employs an improved non-maximum suppression method, drawing upon the traditional Soft-NMS technique, an improved version of non-maximum suppression. Soft-NMS does not directly suppress detection boxes with IoU greater than a threshold; instead, it reduces the impact on subsequent detection boxes by lowering their confidence level. Its implementation process is largely the same as that of non-maximum suppression, except that when the IoU of an overlapping region is determined to be greater than a set threshold, it does not remove it from the list. Instead, it lowers their confidence level based on an adjustable parameter, sorts them from highest to lowest confidence, and repeats this process multiple times. Finally, based on the set confidence threshold, detection boxes with confidence levels higher than the threshold are retained. The operation process is as follows: First, for each category, the target detection results are sorted in descending order based on confidence level. The detection box with the highest confidence level is taken and used as the first element of the retained results list. Next, the remaining detection boxes are compared with the elements in the retained results list in turn. If the IoU (Intersection over Union) of the overlapping area is greater than the set threshold, an adjustable parameter is added to the confidence score. The process continues to select detection boxes with high confidence scores from the remaining detection boxes and repeats the above process. At the end of the traversal of detection boxes, the penalty for confidence scores is increased until the final retained results are the detection boxes with confidence scores higher than the set threshold.

[0114] For example, based on an adjustable parameter (such as alpha; the larger the alpha, the greater the reduction in confidence and the more obvious the suppression effect; the smaller the alpha, the smaller the reduction in confidence and the smoother the suppression effect), the confidence of the bounding boxes is reduced, and then they are sorted from highest to lowest confidence. After repeating the above process multiple times, based on a set confidence threshold, the detection boxes with confidence levels above the threshold are retained. In this way, redundant boxes can be removed, retaining only the most likely target boxes. This ensures that boxes between different objects do not interfere with each other and effectively solves the problem of different objects only generating one detection due to overlap. Finally, the target detection results in the passage space are plotted on the image, completing the passage space target detection. The confidence update formula is as follows:

[0115]

[0116] Soft-NMS has a significant effect on small target detection. However, for the driving scenario involved in the embodiments of the present invention, not all targets to be detected are small targets or overlapping targets. Therefore, Soft-NMS may frequently result in redundant boxes due to insufficient suppression.

[0117] Therefore, in this embodiment of the invention, at the end of the loop that reduces the confidence of detection boxes in Soft-NMS, the penalty for detection boxes with low confidence but not yet reaching the threshold is increased to reduce the occurrence of redundant boxes. The adjustable parameter can be expressed as:

[0118]

[0119] Where, r n R represents the maximum number of iterations. n This indicates the current iteration number, where μ represents a number close to zero.

[0120] Combining equations (16) and (17), we obtain a new confidence update formula:

[0121]

[0122] S190. The spatial target detection results, lane line semantic segmentation results and drivable semantic segmentation results are superimposed using the classification module to obtain the prediction result of the driving image to be perceived.

[0123] In this embodiment of the invention, the output images of different tasks are processed by a post-processing module and finally output as superimposed images of traffic space target detection, lane line semantic segmentation and drivable area semantic segmentation.

[0124] In some embodiments of the present invention, in order to train and optimize the deep learning-based panoramic driving perception network, the present invention needs to select appropriate processing and perform preprocessing.

[0125] In this application, the selection and preprocessing of the dataset need to meet the detection objectives required for autonomous driving, and the dataset should include the following:

[0126] (1) Original images and lane line semantic segmentation labels under different weather conditions, lighting conditions, and road conditions.

[0127] (2) Original images and labels for six types of traffic space detection (object detection task). The labeled instances should include: cars, vehicles, motorcycles, bicycles, pedestrians, traffic signs, etc.

[0128] (3) Original images of different weather, different lighting, and different roads, and semantic segmentation labels of drivable areas.

[0129] In the data preprocessing stage, data augmentation is a crucial method for improving the robustness of object detection algorithms. Data augmentation essentially involves making images more diverse. By processing images, the network can see the same object from multiple angles, preventing predictions from being significantly worse than expected due to changes in the viewing angle of the image.

[0130] For semantic segmentation, the best data preprocessing is to crop a portion of the original image, and similarly crop the label portion. Since the values ​​on the semantic segmentation label map represent the classification of the part of the original image that needs to be segmented, improper operations can easily lead to inaccurate labels. Therefore, image processing should ideally avoid operations such as rotation and scaling. However, to improve the stability of object detection, both the original image and the semantic segmentation and object detection labels can be resized. However, for semantic segmentation tasks, the label values ​​have some distortion. Nevertheless, to improve the robustness of the algorithm, the data preprocessing can perform minor stretching, rotation, translation, and cropping on the image. This part is called radiometric transformation enhancement in deep learning, an effective data augmentation method that can help improve the generalization ability of deep learning models. Additionally, color HSV channel enhancement can be added to the original image. This increases the diversity of training data by transforming the image's color channels, which can also improve the model's generalization ability.

[0131] This patent constructs a deep learning model based on multi-scale convolution and attention mechanisms, and applies it to solve panoramic driving perception tasks in complex environments. It can simultaneously handle three visual perception tasks: target detection in the traffic space, lane line semantic segmentation, and drivable area semantic segmentation, and can run in real-time. This patent uses the YOLOv5 backbone network, with the Spatial Convolutional Pooling Pyramid (ASPP) module and Feature Pyramid Network (FPN) as enhanced feature extraction modules. ASPP generates and fuses features at different scales, while FPN fuses features at different semantic levels. After the FPN fuses the features at different semantic levels generated by the backbone network, a fused feature map of the driving image is obtained. The multi-scale convolution module is then used to extract features from the fused feature map of the driving image, resulting in multi-scale lane line semantic features and multi-scale drivable area semantic features of the driving image. For the self-attention and cross-attention mechanisms, the multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image are input. After passing through the Transformer encoding layer, F1 and F2 are obtained. F1 and F2 are then applied to the axial attention mechanism and then to the cross-attention mechanism to obtain two sets of outputs, F1 and F2. 1 F2 1 Then set F1, F2 and F1 respectively. 1 F2 1 The results are concatenated according to the classification of lane line semantic segmentation task and drivable area semantic segmentation task to obtain two sets of F1 results. 2 and F2 2These will serve as inputs to the lane line semantic segmentation head and the drivable area semantic segmentation head in the decoder, respectively. The decoder includes an object detection head and a semantic segmentation head. The semantic segmentation head includes a drivable area semantic segmentation head and a lane line semantic segmentation head. The object detection head is the same as YOLOv4, employing an anchor-based multi-scale object detection strategy. The lane line semantic segmentation head and the drivable area semantic segmentation head, upon receiving the semantic segmentation feature map F2... 1 F2 2 After three upsampling operations, the semantic segmentation prediction results for lane lines and drivable areas can be obtained respectively.

[0132] Since there are three decoders in the network, the multi-task loss consists of three parts, including the object detection loss L. det These are the weighted sums of the classification loss, target loss, and bounding box loss, as shown in equation (19):

[0133] L det =α1L class +α2L obj +α3L box (19)

[0134] In the formula L class and L obj Focus loss is used to reduce the loss on well-classified samples, thereby forcing the network to focus on difficult-to-classify samples. class Used for penalty classification, L obj Used to indicate the confidence level of a prediction. L box It's L Ciou The method takes into account the distance between the predicted bounding box and the ground truth, the overlap rate, the scale similarity, and the aspect ratio.

[0135] Semantic segmentation of drivable region L da-seg Lane line semantic segmentation L ll-seg All losses include LogitsL ce The cross-entropy loss aims to minimize the classification error between the network's output pixel and the target. The IoU loss is added to L... ll-seg It is particularly effective for predicting sparse categories of lane lines.

[0136]

[0137] The IoU loss calculation process triggers the calculation of the four coordinate points of the BBox, performs IoU on the ground truth, and then concatenates the generated results into a complete code. Since IoU is a scale-invariant representation, it can solve the problem that traditional methods, when calculating the l1 or l2 loss of {x,y,w,h}, will see the loss increase with the scale.

[0138] L da-seg and L ll-seg They are defined as equations (21) and (22) respectively.

[0139] L da-seg =L ce (twenty one)

[0140] L ll-seg =L ce +L Iou (twenty two)

[0141] Therefore, the final loss is a weighted sum of the three parts, as shown in equation (23).

[0142] L all =γ1L det +γ2L da-seg +γ3L ll-seg (twenty three)

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0144] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based panoramic driving perception method, characterized in that, The method includes: Acquire the driving image to be perceived and preprocess the driving image; The driving image is subjected to multidimensional feature extraction using a feature extraction subnetwork to obtain a multidimensional feature map of the driving image; The multidimensional feature map of the driving image is fused using a feature fusion subnetwork to obtain the fused feature map of the driving image; The multi-scale convolution module is used to extract features from the fused feature map of the driving image to obtain multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image. The multi-scale lane line semantic features and multi-scale drivable semantic features of the driving image are processed using a hybrid attention mechanism module to obtain hybrid attention lane line semantic features and hybrid attention drivable semantic features of the driving image. The processing steps include: The axial attention mechanism is used to process the feature vectors of multi-scale lane line semantic features in a wide dimension and the feature vectors of multi-scale drivable semantic features in a wide dimension. The axial attention mechanism is used to process the high-dimensional feature vectors of multi-scale lane line semantic features and the high-dimensional feature vectors of multi-scale drivable semantic features. The cross-attention mechanism is used to process the feature vector of multi-scale lane line semantic features after axial attention in a wide dimension and the feature vector of multi-scale lane line semantic features after axial attention in a high dimension. The cross-attention mechanism is used to process the feature vectors of multi-scale drivable semantic features after axial attention in a wide dimension and the feature vectors of multi-scale drivable semantic features after axial attention in a high dimension. The cross-attention mechanism is used to process the feature vector of multi-scale lane line semantic features after the first cross-attention and the feature vector of multi-scale drivable semantic features after the first cross-attention. The cross-attention mechanism is used to process the high-dimensional feature vector of the multi-scale lane line semantic features after the first cross-attention and the wide-dimensional feature vector of the multi-scale drivable semantic features after the first cross-attention. The lane line semantic segmentation module is used to process the hybrid attention lane line semantic features of the driving image to obtain the lane line semantic segmentation result of the driving image. The drivable region semantic segmentation module is used to perform feature processing on the hybrid attention drivable semantic features of the driving image to obtain the drivable semantic segmentation result of the driving image; The spatial target detection module is used to detect and process the multi-dimensional features of driving images to obtain the spatial target detection results of driving images; The spatial target detection results, lane line semantic segmentation results, and drivable semantic segmentation results are superimposed using the classification module to obtain the prediction result of the driving image to be perceived. The hybrid attention includes axial attention and cross attention.

2. The deep learning-based panoramic driving perception method according to claim 1, characterized in that, The multi-scale convolution module includes multiple convolution kernels of different sizes or multiple dilated convolution kernels of different magnification.

3. The deep learning-based panoramic driving perception method according to claim 1, characterized in that, The axial attention mechanism module includes a first axial attention unit and a second axial attention unit with identical structures, as well as a first cross attention unit and a second cross attention unit with identical structures. Each axial attention unit comprises an axial attention layer, a middle residual connection and normalization layer, a feedforward neural network layer, and a tail residual connection and normalization layer connected sequentially. Each cross attention unit comprises a concatenation connection layer, a cross attention layer, a middle residual connection and normalization layer, a feedforward neural network layer, and a tail residual connection and normalization layer connected sequentially. The tail residual connection and normalization layer of the first axial attention unit is connected to the middle residual connection and normalization layer and the concatenation connection layer of the first cross attention unit, as well as the concatenation connection layer of the second cross attention unit. The tail residual connection and normalization layer of the second axial attention unit is connected to the middle residual connection and normalization layer and the concatenation connection layer of the second cross attention unit.

4. The deep learning-based panoramic driving perception method according to claim 1, characterized in that, The feature vectors of each axial dimension processed by the axial attention mechanism are stacked multiple times with the feature vectors of the cross dimension processed by the cross attention mechanism.

5. The deep learning-based panoramic driving perception method according to claim 4, characterized in that, The stacking process includes using the feature vectors in each axial dimension after processing by the axial attention mechanism as the input feature vectors before processing by the cross attention mechanism, or / and using the feature vectors in the cross dimension after processing by the cross attention mechanism as the input feature vectors before processing by the axial attention mechanism.

6. The deep learning-based panoramic driving perception method according to claim 1, characterized in that, Before obtaining the spatial target detection result of the driving image by detecting and processing the multi-dimensional features of the driving image using the traffic space target detection module, the method further includes obtaining the pixel-level spatial position information of the lane lines from the lane line semantic segmentation result, calculating the midpoint of the lane lines on the same plane and performing curve fitting in the vertical direction, determining the thickness of the fitted curve by calculating the width of the lane lines in the horizontal direction, correcting the horizontal spacing of the output lane lines by multiple pairs based on the prior knowledge that the lane lines are parallel to each other, and drawing the fitted lane lines onto the image to complete the image segmentation of the lane lines.

7. The deep learning-based panoramic driving perception method according to claim 6, characterized in that, The calculation of the midpoints of lane lines on the same plane and the vertical curve fitting process includes traversing the midpoints of the lane lines, assigning the midpoints of lane lines that conform to the current vertical height range to the same baseline according to the traversal order, and assigning the midpoints of lane lines that do not conform to the current vertical height range to the next baseline, until all lane lines have been traversed; taking the midpoint of the leftmost lane line on the same plane as the reference point, taking the baseline containing the reference point as the baseline and forming a lane line fitting segment, sorting the baseline and other baselines according to their vertical height to determine the vertical positional relationship of each baseline; traversing the baselines other than the baseline and searching backwards, splicing the midpoints of adjacent lane lines belonging to the same baseline to form a lane line fitting segment. The process involves fitting lane lines to segments; using a reference point as the root node, the midpoints of the remaining lane lines on the reference line as the first child node of the root node, and the midpoints of the lane lines on other baselines as the second child node of the root node; identifying the relative positions between the midpoints of each lane line according to the positional relationships of the root node, each first child node, and the second child node; determining the relative positional relationships between the lane line fitting segments of the reference line and the lane line fitting segments of the other baselines according to the vertical positional relationships between the reference line and the other baselines; and determining the pixel-level semantic tree of the lane lines according to the relative positions between the midpoints of each lane line; and processing the semantic tree to obtain the semantic fitting result of the lane lines in the vertical direction.

8. The deep learning-based panoramic driving perception method according to claim 1, characterized in that, Before using the traffic space target detection module to detect and process the multi-dimensional features of the driving image and obtain the spatial target detection result of the driving image, the method further includes sorting the target detection results in descending order according to the confidence level; taking out the detection box with the highest confidence level and using it as the first element of the retained result list; The remaining detection boxes are compared with the elements in the retained results list in turn. If the IoU of the overlapping area is greater than the set threshold, an adjustable parameter is added to the confidence score. The process continues to select detection boxes with high confidence scores from the remaining detection boxes and repeats the above process. At the end of the detection box traversal, the penalty for the confidence score is increased until the final retained results are the detection boxes with confidence scores higher than the set threshold.

Images