Autonomous driving scene classification algorithm based on deep learning

Abstract

This invention discloses a deep learning-based autonomous driving scene classification algorithm. The algorithm comprises obtaining autonomous driving scene video data, constructing an input dataset using data preprocessing, constructing a preliminary autonomous driving scene classification model, training the preliminary autonomous driving scene classification model using the constructed input dataset to obtain the autonomous driving scene classification model, and completing the autonomous driving scene classification using the obtained autonomous driving scene classification model. This invention improves the robustness of the video classification algorithm and can effectively solve the autonomous driving road, weather, and vehicle driving motion classification problems. Its classification accuracy and stability surpass those of currently state-of-the-art video classification algorithms.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology in the field of intelligent driving, and specifically relates to an autonomous driving scene classification algorithm based on deep learning. Background Art

[0002] The development of autonomous driving perception technology requires massive amounts of driving image data. The classification accuracy and efficiency of driving image data are crucial factors influencing the development of intelligent driving technology. The primary goal of autonomous driving scene classification is to categorize driving scenarios based on road conditions, weather factors, and vehicle motion patterns. For example, in complex weather conditions such as rain, snow, fog, and haze, categorizing road scenes such as highways, culverts, intersections, ramps, and bridges, as well as categorizing vehicle motion patterns such as straight ahead and turning, facilitates adjustments to vehicle speed and driving mode. Furthermore, accurate scene data classification helps identify abnormal driving conditions. This valuable data helps mitigate potential driving risks. Efficient scene data classification facilitates image data annotation and algorithm testing, facilitating the classification and management of massive amounts of data.

[0003] Among traditional methods for classifying and processing massive amounts of data, high-level semantic scene classification methods use scene images as a foundation, defining main scenes and sub-scenes. Key information is then identified for each sub-scene in parallel based on pre-set parameter boundaries and extraction criteria. Based on pre-defined scene fusion conditions and principles, the sub-scenes are then fused together to produce a multi-dimensional driving scene for classification and recognition. However, the image classification process requires relatively complex solutions, and the diverse representation formats also make the processing time-consuming.

[0004] With the continuous updating of computer technology, road scene classification methods based on traditional deep learning, road scene classification and road segmentation methods based on convolutional neural networks, and image scene classification methods based on attention mechanisms have been proposed one after another, solving the image classification problem to varying degrees. However, due to the limitations of the methods themselves and the complexity of massive image classification processing, the current methods used for massive data classification management cannot achieve the expected ideal results when processing image classification. At the same time, the processing time is relatively long and the cost is relatively high, making them unsuitable for the actual field of autonomous driving technology. Summary of the Invention

[0005] The purpose of this invention is to provide an autonomous driving scene classification algorithm based on deep learning with good versatility, strong robustness and high classification accuracy.

[0006] The deep learning-based autonomous driving scene classification algorithm provided by the present invention includes the following steps:

[0007] S1. Obtain video data of autonomous driving scenarios and construct an input dataset using data preprocessing.

[0008] S2. Build a preliminary model for autonomous driving scenario classification;

[0009] S3. Using the input data set constructed in step S1 to train the preliminary autonomous driving scene classification model constructed in step S2, to obtain an autonomous driving scene classification model;

[0010] S4. Use the autonomous driving scene classification model obtained in step S3 to complete the autonomous driving scene classification.

[0011] The step S1 of obtaining the autonomous driving scene video data and constructing the input data set by data preprocessing specifically includes the following steps:

[0012] Data preprocessing:

[0013] (1) Trimming the autonomous driving scene video data into a single video with a fixed frame number;

[0014] (2) Decoding the cropped video and sampling N images at equal intervals from the video for video classification feature extraction, where N is the number of images and is an integer;

[0015] (3) Perform average pooling on the video image obtained in step (2) and downsample the image size to 1 / 2 of the original image size, denoted as: F = [f1, f2, ..f i .,f N-1 ,f N ], where f i is the size of any image after average pooling;

[0016] (4) For the N video images obtained in step (3), the difference between each two adjacent frames is calculated to obtain the frame difference, and the obtained frame difference is subjected to the maximum pooling operation to highlight the change difference of the image; the maximum pooling process downsamples the size of the frame difference to 1 / 2 of the original frame difference size, which is recorded as: DF = [I1, I2, I3, .. I i .,I N-1 ], where I i is the size of the frame difference of any image after the maximum pooling process;

[0017] (5) The acquired image F = [f1, f2, ..f i .,f N-1 ,f N ] and frame difference DF=[I1,I2,I3,..I i .,I N-1 ] to perform weighted summation processing as follows:

[0018] p1=αf1+βI1

[0019] p2=αf2+βI2

[0020] …

[0021] p N-1 =αf N-1 +βI N-1

[0022] p N =αf N +βI N-1

[0023] Among them, α and β represent the weights of the image and frame difference in the summation process, which must meet the following conditions: 0≤α≤1, 0≤β≤1, and α+β≤1; I i =MaxPool(f i+1 -f i ), MaxPool is the maximum pooling process;

[0024] Construct the input dataset, including:

[0025] (1) Using the preprocessed data, crop n images with a resolution of 224×224 from them, where n represents the number of images and n≤N;

[0026] The cropping method is as follows: cropping each image in a fixed size format; cropping from top to bottom or from left to right;

[0027] If the vertical length of the image is greater than the horizontal length, it is cropped from top to bottom: the first cropping block is cropped from the top of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the bottom of the image;

[0028] If the vertical length of the image is shorter than the horizontal length, cropping is performed from left to right: the first cropping block is cropped from the left end of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the right end of the image;

[0029] If the vertical length of the image is equal to the horizontal length, you can choose to crop from top to bottom or from left to right.

[0030] When cropping, keep the size of each crop block consistent, and at the same time, adjacent crop blocks can overlap during the cropping process;

[0031] (2) The n images obtained in step (1) are divided into image blocks with a resolution of 16×16. The image blocks are converted into vector form by projection and tiling. The vector image blocks are subjected to relative position encoding and category embedding processing. The processed data are constructed into the input data set.

[0032] Step S3 uses the input data set constructed in step S1 to train the preliminary autonomous driving scene classification model constructed in step S2 to obtain the autonomous driving scene classification model, specifically including:

[0033] Construct several spatiotemporal attention perception blocks, so that the output of the previous spatiotemporal perception block is used as the input of the next spatiotemporal perception block, and the input dataset constructed in step S1 is used as the input of the first spatiotemporal perception block to participate in the construction of the first spatiotemporal perception block;

[0034] The spatiotemporal attention perception block includes: multi-head temporal attention module, image block convolutional attention module, multi-head spatial attention module, multi-layer perceptron module, first-layer normalization module, second-layer normalization module, third-layer normalization module and fourth-layer normalization module.

[0035] The workflow of each spatiotemporal attention perception block specifically includes: after the input data is processed by the first-layer normalization module, the output of the first-layer normalization processing is used as the input of the multi-head temporal attention module, and the output of the multi-head temporal attention module is summed with the input data to obtain the first summed output data; the first summed output data is used as the input of the second-layer normalization module, and the output of the second-layer normalization module is used as the input of the image block convolution attention module, and the output of the image block convolution attention module is summed with the first summed output data to obtain the second summed output data; the second summed output data is used as the input of the third-layer normalization module. The output of the third-layer normalization module is used as the input of the multi-head temporal attention module, and the output of the multi-head spatial attention module is summed with the second summed output data to obtain the third summed output data; the third summed output data is used as the input of the fourth-layer normalization module, and the output of the fourth-layer normalization module is used as the input of the multi-layer perceptron, and the output of the multi-layer perceptron is summed with the third summed output data to obtain the fourth summed output data, and the fourth summed output data is used as the output of a single spatiotemporal attention perception block and as the input of the next adjacent spatiotemporal attention perception block to participate in the construction of the next spatiotemporal attention perception block.

[0036] (1) Multi-head temporal attention module:

[0037] Using the input dataset constructed in step S1, temporal information modeling is performed using the temporal self-attention mechanism. Specifically, the following steps are performed: The image with a resolution of 224×224 obtained by processing in step S1 is obtained, and the temporal self-attention mechanism is used to model the video data features in the temporal dimension. Temporal attention is calculated for image blocks of the same color and position in different images. The calculation method is as follows:

[0038] Q=xW q ,K=xW k ,V=xW v

[0039]

[0040] Among them, x is the original image feature of the input; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention;

[0041] Each image is cropped into a square grid of m rows × m columns. The grid in the first row and first column is used for the first calculation, the grid in the first row and second column and the grid in the second row and second column are used for the second calculation, the grid in the fourth row and fourth column is used for the third calculation, and so on, until all grids are calculated.

[0042] (2) Image block convolution attention module: local spatial information modeling is performed through CNN, specifically including:

[0043] By performing convolution operations at the image block level, the static local correlation between image blocks is modeled, and on this basis, the self-attention mechanism is further used to perform dynamic global modeling to encode image features;

[0044] (2-1) First, a convolution block of a set size is used to model static local information at the image block level. By supplementing the image edges in the convolution operation, a feature map with the same size as the original image is obtained.

[0045] (2-2) The feature map is then concatenated with the original image, retaining the common features of the original image and the local information modeling to obtain a feature map of the local modeling;

[0046] (2-3) Based on the spliced ​​image, two layers of 1×1 convolution operations are used to obtain the corresponding weight information for calculating attention;

[0047] (2-4) At the same time, a 1×1 convolutional layer is also used to encode the original image block, and then the encoded image is multiplied by the weight to obtain the calculation result of the attention mechanism;

[0048] (2-5) In order to prevent gradient explosion and network degradation, a residual connection is used to add the feature map obtained in step (2-2) and the calculation result of the attention mechanism obtained in step (2-4) as the output of the module.

[0049] (3) Multi-head spatial attention module:

[0050] Global spatial information modeling is performed through the spatial attention mechanism. Specifically, the following steps are used: position information modeling is used as part of the neural network to learn the positional dependencies between features, and the position and semantic relationships between image blocks are utilized; image blocks are obtained after data preprocessing, and the relative positions between image blocks are modeled. The position information of a single image block is defined as:

[0051]

[0052]

[0053]

[0054] Among them, (Cx i ,Cy i ) is the geometric center of the image block; w i is the width of the image block; h i is the height of the image block; is the pixel position of the upper left corner of the i-th image block, is the pixel position of the lower right corner of the i-th image block; is the horizontal coordinate of the pixel position in the lower right corner; is the vertical coordinate of the pixel position in the lower right corner; is the horizontal coordinate of the pixel position in the upper left corner; is the vertical coordinate of the pixel position in the upper left corner;

[0055] The relative position relationship between the i-th image block and the j-th image block is described by the following calculation method:

[0056]

[0057] λ ij =RELU(ω T FC(r ij ))

[0058] Among them, r ij is the relative position relationship; FC is a fully connected layer neural network used to obtain r ijHigh-dimensional representation of ω; T is the weight parameter to be learned, λ ij is the relative geometric feature; the RELU activation function is used for zero trimming operation, which only retains the case when the relative geometric relationship is calculated as a positive number, so that only the relationship between image blocks with geometric relationships is considered; w i is the width of the i-th image block; w j is the width of the jth image block; h i is the height of the i-th image block; h j is the height of the j-th image block.

[0059] The relative position information calculated by relative position encoding enhances the calculation of attention in Transformer. The calculation method of self-attention is as follows:

[0060] Q=xW q ,K=xW k ,V=xW v

[0061]

[0062] Among them, x is the original image feature of the input; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention.

[0063] Step S4 uses the autonomous driving scene classification model obtained in step S3 to complete the autonomous driving scene classification, specifically including: for the autonomous driving scene classification model obtained in step S3, using a residual connection method to prevent model degradation, solving the scene classification problem by stacking different numbers of spatiotemporal attention perception blocks, and finally outputting the classification results through a fully connected layer.

[0064] The deep learning-based autonomous driving scene classification algorithm provided by the present invention processes the video data of the autonomous driving scene by adopting a data preprocessing method, constructs an input data set using the preprocessed data, and obtains the autonomous driving scene classification model by training the preliminary autonomous driving scene classification model, thereby completing the classification processing of the autonomous driving scene. The present invention has good versatility, strong robustness and high classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the process flow of the present invention.

[0066] Figure 2 Flowchart of the autonomous driving scene classification algorithm.

[0067] Figure 3 Schematic diagram of the frame difference calculation and image feature fusion process.

[0068] Figure 4 Schematic diagram of the image cropping method for autonomous driving scenes.

[0069] Figure 5 Schematic diagram of the image cropping method for autonomous driving scenes.

[0070] Figure 6 Schematic diagram of Transformer encoder data input.

[0071] Figure 7 Schematic diagram of the calculation method of the temporal self-attention mechanism.

[0072] Figure 8 Schematic diagram of the self-attention mechanism with the ability to model local information.

[0073] Figure 9 Schematic diagram of the computational flow of the self-attention mechanism with the ability to model local information.

[0074] Figure 10 A schematic diagram of an autonomous driving intersection scenario.

[0075] Figure 11 Schematic diagram of the autonomous driving bridge scenario.

[0076] Figure 12 Schematic diagram of the toll booth and ramp scene.

[0077] Figure 13 Schematic diagram of vehicle turning and congestion scenarios. DETAILED DESCRIPTION

[0078] like Figure 1 The figure shows a flow chart of the method of the present invention: The deep learning-based autonomous driving scene classification algorithm provided by the present invention includes the following steps:

[0079] like Figure 2 The figure shows the flow chart of the autonomous driving scene classification algorithm, which mainly includes four parts: video data processing and loading, image segmentation and relative position encoding, spatial self-attention mechanism with local information modeling capability, and temporal self-attention mechanism enhanced by video frame difference.

[0080] S1. Obtain video data of autonomous driving scenarios and construct an input dataset using data preprocessing. This includes:

[0081] Data preprocessing:

[0082] (1) Trimming the autonomous driving scene video data into a single video with a fixed frame number;

[0083] (2) Decoding the cropped video and sampling N images at equal intervals from the video for video classification feature extraction, where N is the number of images, which is an integer ranging from 8 to 16;

[0084] (3) Perform average pooling on the video image obtained in step (2) and downsample the image size to 1 / 2 of the original image size, denoted as: F = [f1, f2, ..f i .,f N-1 ,f N ], where f i is the size of any image after average pooling;

[0085] (4) For the N video images obtained in step (2), perform difference processing on each two adjacent frames, and perform maximum pooling operation on the obtained frame difference to highlight the change difference of the image. The maximum pooling process downsamples the size of the frame difference to 1 / 2 of the original frame difference size, which is recorded as: DF = [I1, I2, I3, .. I i .,I N-1 ], where I i is the size of the frame difference of any image after maximum pooling processing;

[0086] The present invention performs average pooling on a video image with a resolution of 1920×1080 and selects a fixed block of 16×16 size to crop the video image;

[0087] (5) The acquired image F = [f1,f2,..fi.,f N-1 ,f N ] and frame difference DF=[I1,I2,I3,..I i .,I N-1 ] to perform weighted summation processing as follows:

[0088] p1=αf1+βI1

[0089] p2=αf2+βI2

[0090] …

[0091] p N-1 =αf N-1 +βI N-1

[0092] p N =αf N +βI N-1

[0093] Among them, α and β represent the weights of the image and frame difference in the summation process, which must meet the following conditions: 0≤α≤1, 0≤β≤1, and α+β≤1; I i =MaxPool(f i+1 -f i ), MaxPool is the maximum pooling process;

[0094] like Figure 2 Figure (a) shown is a flowchart of the autonomous driving scene classification algorithm. As shown in the flowchart, the calculation input of the temporal attention mechanism incorporates the frame difference information of two adjacent frames of images. The maximum pooled frame difference information highlights the characteristics of the image, further enhancing the temporal attention's ability to capture changes in video images.

[0095] like Figure 2 Figure (b) shows a schematic diagram of the spatiotemporal attention perception block. This method proposes a spatiotemporal attention perception block for autonomous driving scene classification tasks, which is used for feature extraction and modeling of video data. To further improve the model's representational capabilities, multi-head temporal attention and multi-head spatial attention are employed, and residual connections are used between different modules to prevent model degradation.

[0096] like Figure 3 The figure shows the process of frame difference calculation and image feature fusion, and the weighted summation of the acquired image F and frame difference DF is referenced. Figure 3 The process shown.

[0097] This method uses a 224×224 resolution window to perform vertical sliding cropping on the image; Figure 4 The following is a schematic diagram of the image cropping method for autonomous driving scenes. Figure 4 way;

[0098] Construct the input dataset, including:

[0099] (1) Using the preprocessed data, crop n images with a resolution of 224×224 from them, where n represents the number of images and n≤N;

[0100] The cropping method is as follows: cropping each image in a fixed size format; cropping from top to bottom or from left to right;

[0101] If the vertical length of the image is greater than the horizontal length, it is cropped from top to bottom: the first cropping block is cropped from the top of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the bottom of the image;

[0102] If the vertical length of the image is shorter than the horizontal length, cropping is performed from left to right: the first cropping block is cropped from the left end of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the right end of the image;

[0103] If the vertical length of the image is equal to the horizontal length, you can choose to crop from top to bottom or from left to right.

[0104] When cropping, keep the size of each crop block consistent, and at the same time, adjacent crop blocks can overlap during the cropping process;

[0105] The method of the present invention uses a fixed format of 90 frames of images to crop a single video;

[0106] (2) The n images obtained in step (1) are divided into image blocks with a resolution of 16×16. The image blocks are converted into vector form by projection and tiling. The vector image blocks are subjected to relative position encoding and category embedding processing. The processed data are constructed into the input data set.

[0107] like Figure 5 The figure shows a schematic diagram of the image cropping method for autonomous driving scenes. This method divides a 224×224 resolution image into image blocks of 16×16 resolution. Figure 6 The figure shows a schematic diagram of the Transformer encoder data input. The three images are divided into image blocks with a resolution of 16×16. The image blocks are converted into vector form using projection and tiling. The vector image blocks are then subjected to relative position encoding and category embedding processing. This is used as the input of the Transformer encoder to obtain 14×14 image blocks.

[0108] S2. Build a preliminary model for autonomous driving scenario classification;

[0109] S3. Using the input data set constructed in step S1, the preliminary autonomous driving scene classification model constructed in step S2 is trained to obtain an autonomous driving scene classification model; specifically, the following steps are performed:

[0110] Construct multiple spatiotemporal attention perception blocks, so that the output of the previous spatiotemporal perception block is used as the input of the next spatiotemporal perception block, and the input dataset constructed in step S1 is used as the input of the first spatiotemporal perception block to participate in the construction of the first spatiotemporal perception block;

[0111] The spatiotemporal attention perception block includes: multi-head temporal attention module, image block convolutional attention module, multi-head spatial attention module, multi-layer perceptron module, first-layer normalization module, second-layer normalization module, third-layer normalization module and fourth-layer normalization module.

[0112] The workflow of each spatiotemporal attention perception block specifically includes: after the input data is processed by the first-layer normalization module, the output of the first-layer normalization processing is used as the input of the multi-head temporal attention module, and the output of the multi-head temporal attention module is summed with the input data to obtain the first summed output data; the first summed output data is used as the input of the second-layer normalization module, and the output of the second-layer normalization module is used as the input of the image block convolution attention module, and the output of the image block convolution attention module is summed with the first summed output data to obtain the second summed output data; the second summed output data is used as the input of the third-layer normalization module. The output of the third-layer normalization module is used as the input of the multi-head temporal attention module, and the output of the multi-head spatial attention module is summed with the second summed output data to obtain the third summed output data; the third summed output data is used as the input of the fourth-layer normalization module, and the output of the fourth-layer normalization module is used as the input of the multi-layer perceptron, and the output of the multi-layer perceptron is summed with the third summed output data to obtain the fourth summed output data, and the fourth summed output data is used as the output of a single spatiotemporal attention perception block and as the input of the next adjacent spatiotemporal attention perception block to participate in the construction of the next spatiotemporal attention perception block.

[0113] (1) Multi-head temporal attention module:

[0114] Using the input dataset constructed in step S1, temporal information modeling is performed using the temporal self-attention mechanism. Specifically, the following steps are performed: The image with a resolution of 224×224 obtained by processing in step S1 is obtained, and the temporal self-attention mechanism is used to fully model the video data features in the temporal dimension. Temporal attention is calculated for image blocks of the same color and position in different images. The calculation method is as follows:

[0115] Q=xW q ,K=xW k ,V=xW v

[0116]

[0117] Among them, x is the original image feature of the input; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention;

[0118] Each image is cropped into a square grid of m rows × m columns. The grid in the first row and first column is used for the first calculation, the grid in the first row and second column and the grid in the second row and second column are used for the second calculation, the grid in the fourth row and fourth column is used for the third calculation, and so on, until all grids are calculated.

[0119] This method calculates sparse temporal attention based on the divided image blocks. Figure 7 The figure shows a schematic diagram of the calculation method of the temporal self-attention mechanism. The corresponding temporal attention is calculated according to the area indicated by the color image block in the figure. The temporal attention of the area without the color image block is calculated by analogy with the color block shown in the figure.

[0120] (2) Image block convolution attention module: local spatial information modeling is performed through CNN, specifically including:

[0121] By performing convolution operations at the image block level, the static local correlation between image blocks is modeled. On this basis, the self-attention mechanism is further used to perform dynamic global modeling to fully encode image features.

[0122] (2-1) First, a convolution block of a set size is used to model static local information at the image block level. By supplementing the image edges in the convolution operation, a feature map with the same size as the original image is obtained.

[0123] (2-2) The feature map is then concatenated with the original image, retaining the common features of the original image and the local information modeling to obtain a feature map of the local modeling;

[0124] (2-3) Based on the spliced ​​image, two layers of 1×1 convolution operations are used to obtain the corresponding weight information for calculating attention;

[0125] (2-4) At the same time, a 1×1 convolutional layer is also used to encode the original image block, and then the encoded image is multiplied by the weight to obtain the calculation result of the attention mechanism;

[0126] (2-5) In order to prevent gradient explosion and network degradation, a residual connection is used to add the feature map obtained in step (2-2) and the calculation result of the attention mechanism obtained in step (2-4) as the output of the module.

[0127] like Figure 8 The following is a schematic diagram of the self-attention mechanism with the ability to model local information. Figure 9 The figure shows a schematic diagram of the computational process of the self-attention mechanism with local information modeling capabilities, using convolutional blocks of size such as 3×3 to perform static local information modeling at the image block level.

[0128] (3) Multi-head spatial attention module:

[0129] Global spatial information modeling is performed through the spatial attention mechanism. Specifically, the following steps are used: position information modeling is used as part of the neural network to learn the positional dependencies between features, and the position and semantic relationships between image blocks are utilized; image blocks are obtained after data preprocessing, and the relative positions between image blocks are modeled. The position information of a single image block is defined as:

[0130]

[0131]

[0132]

[0133] Among them, (Cx i ,Cy i ) is the geometric center of the image block; w i is the width of the image block; h i is the height of the image block; is the pixel position of the upper left corner of the i-th image block, is the pixel position of the lower right corner of the i-th image block; is the horizontal coordinate of the pixel position in the lower right corner; is the vertical coordinate of the pixel position in the lower right corner; is the horizontal coordinate of the pixel position in the upper left corner; is the vertical coordinate of the pixel position in the upper left corner;

[0134] The relative position relationship between the i-th image block and the j-th image block is described by the following calculation method:

[0135]

[0136] λ ij =RELU(ω T FC(r ij ))

[0137] Among them, r ij is the relative position relationship; FC is a fully connected layer neural network used to obtain r ij High-dimensional representation of ω; T is the weight parameter to be learned, λ ij is the relative geometric feature; the RELU activation function is used for zero trimming operation, which only retains the case when the relative geometric relationship is calculated as a positive number, so that only the relationship between image blocks with geometric relationships is considered; w i is the width of the i-th image block; w j is the width of the jth image block; h i is the height of the i-th image block; h jis the height of the j-th image block.

[0138] The relative position information calculated by relative position encoding enhances the calculation of attention in Transformer. The calculation method of self-attention is as follows:

[0139] Q=xW q ,K=xW k ,V=xW v

[0140]

[0141] Among them, x is the original image feature of the input; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention.

[0142] S4. Use the autonomous driving scene classification model obtained in step S3 to complete the autonomous driving scene classification; specifically, for the autonomous driving scene classification model obtained in step S3, use a residual connection method to prevent model degradation, solve the scene classification problem by stacking different numbers of spatiotemporal attention perception blocks, and finally output the classification results through a fully connected layer.

[0143] like Figure 10 The following is a schematic diagram of an autonomous driving intersection scene. In the actual application of autonomous driving scene classification, Figure 10 Taking the intersection scene shown as an example, thanks to the spatiotemporal attention mechanism, the scene classification algorithm proposed in the present invention can accurately classify data in different weather and different intersection scenes.

[0144] like Figure 11 The following is a schematic diagram of the autonomous driving bridge scenario. Figure 11 Taking the bridge scene shown as an example, thanks to the modeling capability of the spatiotemporal attention perception block, the scene classification algorithm proposed in this invention can accurately classify distant small target scenes.

[0145] like Figure 12 The diagram shows a toll booth and ramp scenario. Figure 12 Taking the toll station and highway ramp scenes shown as an example, the scene classification algorithm proposed in the present invention can eliminate the interference of buildings, billboards, roads, and bridges, and accurately characterize the characteristics of toll stations and ramps.

[0146] like Figure 13The diagram shows a vehicle turning and congestion scenario. Figure 13 Taking the vehicle turning action and vehicle congestion scene shown as examples, the scene classification algorithm proposed in the present invention can model and accurately classify the vehicle's action behavior.

Claims

1. A deep learning-based autonomous driving scene classification algorithm, comprising the following steps: S1. Obtain video data of autonomous driving scenarios and construct an input dataset using data preprocessing. S2. Build a preliminary model for autonomous driving scenario classification; S3. Using the input data set constructed in step S1, the preliminary autonomous driving scene classification model constructed in step S2 is trained to obtain an autonomous driving scene classification model; specifically, the following steps are performed: Construct several spatiotemporal attention perception blocks, so that the output of the previous spatiotemporal perception block is used as the input of the next spatiotemporal perception block, and the input dataset constructed in step S1 is used as the input of the first spatiotemporal perception block to participate in the construction of the first spatiotemporal perception block; The spatiotemporal attention perception block includes: multi-head temporal attention module, image block convolutional attention module, multi-head spatial attention module, multi-layer perceptron module, first layer normalization module, second layer normalization module, third layer normalization module and fourth layer normalization module; The workflow of each spatiotemporal attention perception block specifically includes: after the input data is processed by the first-layer normalization module, the output of the first-layer normalization processing is used as the input of the multi-head temporal attention module, and the output of the multi-head temporal attention module is summed with the input data to obtain the first summed output data; the first summed output data is used as the input of the second-layer normalization module, and the output of the second-layer normalization module is used as the input of the image block convolution attention module, and the output of the image block convolution attention module is summed with the first summed output data to obtain the second summed output data; the second summed output data is used as the input of the third-layer normalization module. The output of the third-layer normalization module is used as the input of the multi-head temporal attention module, and the output of the multi-head spatial attention module is summed with the second summed output data to obtain third summed output data; the third summed output data is used as the input of the fourth-layer normalization module, and the output of the fourth-layer normalization module is used as the input of the multi-layer perceptron, and the output of the multi-layer perceptron is summed with the third summed output data to obtain fourth summed output data, and the fourth summed output data is used as the output of a single spatiotemporal attention perception block and as the input of the next adjacent spatiotemporal attention perception block to participate in the construction of the next spatiotemporal attention perception block; S4. Use the autonomous driving scene classification model obtained in step S3 to complete the autonomous driving scene classification.

2. The deep learning-based autonomous driving scene classification algorithm according to claim 1 is characterized in that The step S1 of obtaining the autonomous driving scene video data and constructing the input data set by data preprocessing specifically includes the following steps: Data preprocessing: (1) Trimming the autonomous driving scene video data into a single video with a fixed frame number; (2) Decoding the cropped video and sampling N images at equal intervals from the video for video classification feature extraction, where N is the number of images and is an integer; (3) Perform average pooling on the video image obtained in step (2) and downsample the image size to 1 / 2 of the original image size, denoted as: F = [f1, f2, ..f i .,f N-1 ,f N ], where f i is the size of any image after average pooling; (4) For the N video images obtained in step (3), the difference between each two adjacent frames is calculated to obtain the frame difference, and the obtained frame difference is subjected to the maximum pooling operation to highlight the change difference of the image; the maximum pooling process downsamples the size of the frame difference to 1 / 2 of the original frame difference size, which is recorded as: DF = [I1, I2, I3, .. I i .,I N-1 ], where I i is the size of the frame difference of any image after maximum pooling processing; (5) The acquired image F = [f1, f2, ..f i .,f N-1 ,f N ] and frame difference DF=[I1,I2,I3,..I i .,I N-1 ] to perform weighted summation processing as follows: p1=αf1+βI1 p2=αf2+βI2 …… p N-1 =αf N-1 +βI N-1 p N =αf N +βI N-1 Among them, α and β represent the weights of the image and frame difference in the summation process, which must meet the following conditions: 0≤α≤1, 0≤β≤1, and α+β≤1; I i =MaxPool(f i+1 -f i ), MaxPool is the maximum pooling process; Construct the input dataset, including: (1) Using the preprocessed data, crop n images with a resolution of 224×224 from them, where n represents the number of images and n≤N; The cropping method is as follows: cropping each image in a fixed size format; cropping from top to bottom or from left to right; If the vertical length of the image is greater than the horizontal length, it is cropped from top to bottom: the first cropping block is cropped from the top of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the bottom of the image; If the vertical length of the image is shorter than the horizontal length, cropping is performed from left to right: the first cropping block is cropped from the left end of the image, the second cropping block is cropped after the first cropping block, and so on until the last cropping block is cropped to the right end of the image; If the vertical length of the image is equal to the horizontal length, you can choose to crop from top to bottom or from left to right. When cropping, keep the size of each crop block consistent, and at the same time, adjacent crop blocks can overlap during the cropping process; (2) The n images obtained in step (1) are divided into image blocks with a resolution of 16×16. The image blocks are converted into vector form by projection and tiling. The vector image blocks are subjected to relative position encoding and category embedding processing. The processed data are constructed into the input data set.

3. The deep learning-based autonomous driving scene classification algorithm according to claim 1, characterized in that The multi-head temporal attention module specifically includes: Using the input dataset constructed in step S1, temporal information modeling is performed using the temporal self-attention mechanism. Specifically, the following steps are performed: obtaining the image with a resolution of 224×224 obtained by processing in step S1, and using the temporal self-attention mechanism to model the video data features in the temporal dimension; calculating the temporal attention for image blocks of the same color and position in different images, using the following calculation method: Q=xW q ,K=xW k ,V=xW v Among them, x is the input original image feature; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention; Each image is cropped into a square grid of m rows × m columns. The grid in the first row and first column is used for the first calculation, the grid in the first row and second column and the grid in the second row and second column are used for the second calculation, the grid in the fourth row and fourth column is used for the third calculation, and so on, until all grids are calculated.

4. The deep learning-based autonomous driving scene classification algorithm according to claim 3 is characterized in that The image block convolution attention module specifically includes: Local spatial information modeling through CNN: By performing convolution operations at the image block level, the static local correlation between image blocks is modeled, and on this basis, the self-attention mechanism is further used to perform dynamic global modeling to encode image features; (2-1) First, a convolution block of a set size is used to model static local information at the image block level. By supplementing the image edges in the convolution operation, a feature map with the same size as the original image is obtained. (2-2) The feature map is then concatenated with the original image, retaining the common features of the original image and the local information modeling to obtain a feature map of the local modeling; (2-3) Based on the spliced ​​image, two layers of 1×1 convolution operations are used to obtain the corresponding weight information for calculating attention; (2-4) At the same time, a 1×1 convolutional layer is also used to encode the original image block, and then the encoded image is multiplied by the weight to obtain the calculation result of the attention mechanism; (2-5) In order to prevent gradient explosion and network degradation, a residual connection is used to add the feature map obtained in step (2-2) and the calculation result of the attention mechanism obtained in step (2-4) as the output of the module.

5. The deep learning-based autonomous driving scene classification algorithm according to claim 4 is characterized in that The multi-head spatial attention module specifically includes: Global spatial information modeling is performed through the spatial attention mechanism. Specifically, the following steps are used: position information modeling is used as part of the neural network to learn the positional dependencies between features, and the position and semantic relationships between image blocks are utilized; image blocks are obtained after data preprocessing, and the relative positions between image blocks are modeled. The position information of a single image block is defined as: Among them, (Cx i ,Cy i ) is the geometric center of the image block; w i is the width of the image block; h i is the height of the image block; is the pixel position of the upper left corner of the i-th image block, is the pixel position of the lower right corner of the i-th image block; is the horizontal coordinate of the pixel position in the lower right corner; is the vertical coordinate of the pixel position in the lower right corner; is the horizontal coordinate of the pixel position in the upper left corner; The vertical coordinate of the pixel position in the upper left corner; The relative position relationship between the i-th image block and the j-th image block is described by the following calculation method: l ij =RELU(ω T FC(r ij )) Among them, r ij is the relative position relationship; FC is a fully connected layer neural network used to obtain r ij High-dimensional representation of ω; T is the weight parameter to be learned, λ ij is the relative geometric feature; the RELU activation function is used for zero trimming operation, which only retains the case when the relative geometric relationship is calculated as a positive number, so that only the relationship between image blocks with geometric relationships is considered; w i is the width of the i-th image block; w j is the width of the jth image block; h i is the height of the i-th image block; h j is the height of the j-th image block; The relative position information calculated by relative position encoding enhances the calculation of attention in Transformer. The calculation method of self-attention is as follows: Q=xW q ,K=xW k ,V=xW v Among them, x is the input original image feature; W q , W k and W v is the learnable parameter matrix; Q is the query after the parameter matrix mapping; K is the key after the parameter matrix mapping; V is the value after the parameter matrix mapping; d k is the square of the query and key vector dimensions; λ is the weight used to obtain the bias; z is the calculated self-attention.

6. The deep learning-based autonomous driving scene classification algorithm according to claim 5, characterized in that Step S4 uses the autonomous driving scene classification model obtained in step S3 to complete the autonomous driving scene classification, specifically including: for the autonomous driving scene classification model obtained in step S3, using a residual connection method to prevent model degradation, solving the scene classification problem by stacking different numbers of spatiotemporal attention perception blocks, and finally outputting the classification results through a fully connected layer.

Images