Method for constructing complex road surface rapid identification model

Abstract

A method for constructing a complex road surface rapid identification model, which is a structure reparameterization and adaptive attention lightweight network, is used as a vehicle front road surface type identification model to identify the road surface in front of the vehicle. The steps are: a feature extraction backbone network is constructed with horizontal, vertical, square, point-shaped multi-branch heterogeneous convolution as the core; a lightweight and efficient attention mechanism is used to design an adaptive channel attention module to focus on high correlation features, and to adaptively aggregate spatial context information according to the feature size, and to adaptively perform local cross-channel interaction according to the feature dimension, so that the model focuses on the features with high road surface correlation; a structure reparameterization method is introduced to decouple the training period and the inference period of the model, and the A 2 Rep-Net model is used to realize the precision and speed trade-off in the road surface type identification task. The present application achieves a good balance between accuracy, real-time performance and lightweight, has high adaptability to complex and variable scenes, and has obvious use effect.

Description

TECHNICAL FIELD

[0001] The application relates to a construction method of a road surface rapid identification model, in particular to a construction method of a complex road surface rapid identification model. BACKGROUND

[0002] With the increasing requirements of driving safety and riding comfort, vehicle active control systems are paid more and more attention. At present, most of the vehicle active control systems are started after the driver's reaction and the vehicle dynamic parameters are estimated, and the time left for the actuator to adjust is very short. For a vehicle in motion, different types of road surfaces represent different unevenness and adhesion coefficients, which are key factors directly affecting the stability of vehicle body control. If the type of the road surface ahead can be known in advance, the vehicle active control system will have more reaction time, which is beneficial to improve the comfort and safety of driving. Therefore, road surface type identification is a key link to realize vehicle active control, and can also provide an important basis for subsequent driving decision and path planning of the vehicle.

[0003] In recent years, researchers have carried out a lot of research on road surface type identification tasks and have achieved considerable results. Traditional road surface type identification methods mainly include adhesion coefficient observation based on dynamic modeling and unevenness estimation based on vibration response. Although the method based on dynamic modeling is relatively accurate and reliable, it cannot guarantee the real-time performance of road surface identification. The method based on vibration response has good real-time performance, but it can only identify the road surface type directly below the vehicle in motion, and cannot predict the road surface type in front of the vehicle. With the rapid development of image processing algorithms and artificial intelligence technology, vehicle vision has become an important means of road surface detection in front of the vehicle due to its advantages of pre-foresight, non-contact and real-time. The road surface type identification method based on vision can be mainly divided into traditional machine learning and deep learning. The traditional machine learning method relies on manual construction and extraction of road surface features, and distinguishes different types of road surfaces through a classifier. However, the expression ability of manually constructed features is limited, and it is difficult to effectively identify complex road conditions, so the application has great limitations. In the past decade, deep learning has achieved better performance than traditional machine learning in image classification, including road surface type identification, due to its deeper feature extraction capability and stronger model robustness. At present, the road surface type identification method based on deep learning mostly uses a relatively complex algorithm architecture (such as deepening or widening the network) to ensure the accuracy of identification. Although it has a high precision gain, the complex architecture will sacrifice the time efficiency of the algorithm and cause a sharp increase in the number of parameters, making it difficult to deploy the model on the vehicle-level chip with limited computing resources. At the same time, factors such as bad weather, light changes and complex backgrounds will also introduce great interference. Therefore, accurate and rapid road surface type identification is still a very challenging task.

[0004] The related patent document CN116311134A discloses a road surface mark identification model training method, an identification method, an apparatus and equipment, the method comprising: obtaining a first sample set and a second sample set, the second sample set comprising road surface images subjected to data enhancement; inputting the first sample set and the second sample set into a preset to-be-trained road surface mark identification model for training, adopting a cross-entropy function as a main loss function, adopting a first loss function to calculate a first error coefficient of a first error type of a result of identification of a target road surface mark by the to-be-trained road surface mark identification model, and adopting a second loss function to calculate a second error coefficient of a second error type of the result of identification of the target road surface mark by the to-be-trained road surface mark identification model; and performing weighted calculation on the three loss functions, and obtaining a road surface mark identification model when a weighted loss coefficient is within a preset threshold range. The above technology has no guiding significance for the following technical problems to be solved by the present application. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a complex road surface rapid identification model construction method, which can effectively identify road surface types in complex driving environments, and the model achieves a good balance of accuracy, real-time performance and lightweight, has high adaptability to complex and variable scenes, and has obvious use effect in road surface type identification tasks in front of the vehicle. To solve the problem that the existing method cannot balance accuracy and speed and is difficult to deploy on the vehicle end.

[0006] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0007] A complex road surface rapid identification model construction method, which is a lightweight network with structure reparameterization and adaptive attention, is used as an identification model of road surface types in front of a vehicle to quickly and accurately identify the road surface in front of the vehicle. The technical solution of the complex road surface rapid identification model construction method comprises the following process steps:

[0008] ① A feature extraction backbone network is constructed with horizontal, vertical, square and point-shaped multi-branch heterogeneous convolution as the core, that is, a multi-branch heterogeneous convolution kernel MHK is constructed (to ensure sufficient extraction of road surface features) to perform multi-branch learning;

[0009] ② A lightweight and efficient attention mechanism is adopted, and an adaptive channel attention module ACA is designed to focus on high-correlation features, adaptively aggregate spatial context information according to feature sizes, and adaptively perform local cross-channel interaction according to feature dimensions. Through the calibration of feature responses between channels, the model focuses on features with high correlation to road surfaces;

[0010] ③Introduce structural reparameterization method to decouple the training period and inference period of the model (network), obtain high-dimensional feature representation through the above multi-branch learning during training, and equivalently convert the multi-branch structure into straight-laid single-branch structure during inference, obtain a lightweight deployment model and inference speed without reducing the performance of the model, that is, introduce structural reparameterization strategy (Rep) to equivalently convert the multi-branch MHK module into a single-branch RepMHK module to obtain a lightweight and efficient model inference, on this basis, A 2 Rep-Net model.

[0011] Need to explain the above technical features is that the real driving scene contains complex background and foreground such as sky, vehicle, pedestrian, building, etc., which will introduce a lot of redundant information to interfere with the identification of the road surface, so the model requires high road feature representation ability and information focusing ability. At the same time, the model needs to be lightweight and efficient to facilitate deployment on car-grade chips. In view of this, a multi-branch heterogeneous convolution kernel (Multi-branch Heterogeneous Kernel, MHK) is constructed to ensure sufficient extraction of road surface features, an adaptive channel attention module (Adaptive Channel Attention, ACA) is designed to focus on high correlation features, and a structural reparameterization strategy is introduced to equivalently convert the multi-branch MHK module into a single-branch RepMHK module to realize lightweight and efficient model inference. On this basis, A 2 Rep-Net model, which achieves a good trade-off between accuracy and speed in the road surface type identification task.

[0012] In the above technical solution, the preferred technical solution can be that the road surface type is one of complex road surfaces such as asphalt road surface, cement road surface, ice and snow road surface, sandy road surface, brick road surface, stone road surface and wet asphalt (wet and slippery) road surface. For A 2 Rep-Net overall architecture (training period), the above A 2The Rep-Net model is a 5-stage convolutional neural network architecture, and three parameters in the module are the channel number, height and width of the input feature map, respectively; the 0th stage is an input stage, which is composed of 1 Stem1 module and 1 ACA module, wherein the Stem1 module is composed of 1 4x4 convolution branch and 1 1x1 convolution branch, and the step is 2, which is used to down-sample the input image and expand the channel dimension, so as to reduce the inherent redundant information of the natural image and preliminarily extract the features; the ACA module uses the attention mechanism to adaptively focus on the features in the channel level, and the specific structure is described below. The 1st to 4th stages are main feature extraction stages, which are composed of 1 Stem2 module, a plurality of MHK modules and 1 ACA module, the Stem2 module is the input module of each stage, which first uses a 1x1 convolution with a step of 2 to down-sample and expand the channel dimension, and then uses a lightweight 3x3 deep convolution (denoted as D3x3) to extract the road surface features, the 3x3 deep convolution is denoted as D3x3, and a constant connection branch is set to avoid the problem of gradient disappearance, the MHK module is the main feature extraction module of each stage, which is composed of four heterogeneous convolution branches of square 3x3, horizontal 1x3, vertical 3x1 and point 1x1, and a constant connection branch in parallel, the multiple convolution kernels of different sizes can make the model have a multi-scale receptive field, and the asymmetric convolution kernel can make the model have robustness to rotation distortion, and the overall road surface feature representation ability of the model is improved. The number of the above MHK modules in the 1st to 4th stages is 2, 4, Θ, 1 respectively, and the depth of the model is adjusted by changing the number of MHK stacking Θ in the 3rd stage; at the end of the 1st to 4th stages, the ACA module is also embedded to adaptively focus on the features, and with the deepening of the network, the feature map size is continuously reduced, in order to ensure that the model can fully learn the road surface features in a small scale space, a channel scaling coefficient Φ is set in the 4th stage to increase the feature dimension and obtain a high-dimensional feature representation, and after the 4th stage, the network uses a layer of full connection and a Softmax function to output the prediction result of the road surface type.

[0013] In the above technical solution, the preferred technical solution can also be that, for the ACA module (training period), in order to optimize the output feature map of each stage of the model, so that it focuses more on the features with high relevance to the road surface, the above ACA module has adaptivity in spatial aggregation and channel interaction, as shown in Figure 2 The ACA module is an adaptive spatial aggregation module, for the input feature map F C×H×W , C is the channel number, H and W are the height and width of the feature map respectively, the ACA module first uses two global pooling branches to aggregate the spatial context information: one is to compress the feature map by global average pooling (GAP) to retain the average response feature FA C×1×1 , i.e. the global overall information embodied in spatial context; the other is to compress the feature map through global max pooling (GMP) to retain the maximum response feature F M C×1×1 , i.e. to emphasize the local prominent details in spatial context, then the ACA module fuses the outputs of the two global pooling branches; considering that the local details contained in the feature map are more abundant when the feature map size is large, in order to fully capture the detailed information in the context in the spatial aggregation process, a weight coefficient β is set for the GMP branch, and the fusion weight of the GMP branch is adaptively adjusted according to the feature map size [H, W], i.e. there is a mapping f(β) between [H, W] and β, if a linear mapping is used, i.e. f = m × β - n, then the expression capacity is limited, in order to realize the adaptive fusion of nonlinearity, and in view of the fact that the feature map size is mostly a power of 2, the average value of H and W is taken as the feature size, and:

[0014]

[0015] Therefore, for a given feature map size [H, W], the proportion of the GMP branch is adaptively obtained as follows:

[0016]

[0017] In formula (1) and formula (2), m and n are parameters, and m = 40 and n = 1 are taken, through the nonlinear mapping, the large size feature map will obtain a higher GMP branch weight, it should be noted that the value of β is always less than 1, which means that the proportion of the GAP branch in the ACA module is always higher than that of the GMP branch, i.e. the global overall information of the feature map always dominates in the aggregation of spatial context. The above ACA module is an adaptive channel interaction module, and after adaptive spatial aggregation, a feature map F C×1×1 is obtained. A C×1×1 + β × F M C×1×1 , the ACA module further performs adaptive channel calibration on the aggregated feature F C×1×1 to focus on the features related to the road surface height, this part cancels the channel dimension reduction operation to obtain more effective channel attention, and uses a one-dimensional convolution 1 × k with a size of k to quickly obtain local cross-channel interaction, where the parameter k is adaptively obtained from the input channel number C, considering that the channel number is also a power of 2, then:

[0018] C = g(k) = 2p × k - q (3)

[0019] Therefore, for a given channel number C, the value of k is adaptively obtained as follows:

[0020]

[0021] In formula (3) and formula (4), p and q are parameters, and are respectively 2 and 1; considering that the size of a convolution kernel is usually an odd number, an absolute value is used here odd Taking the nearest odd number upward, the feature map F C×1×1 After adjusting the channel weight by 1xk convolution, a normalized channel attention map A is generated through a SiLU activation function C×1×1 , the original input feature map F C×H×W is multiplied by the channel attention map A C×1×1 , and point multiplication fusion is performed, so that the feature map F' recalibrated by the ACA module is finally obtained C×H×W . The ACA module is embedded at the end of each stage of the network to adaptively calibrate the feature map, strengthen the features highly related to the road type, suppress the redundant features with low correlation, and improve the classification and discrimination ability of the model for complex roads.

[0022] In the above technical solution, the preferred technical solution can also be that, for structure reparameterization (inference period), in the training period, A 2 The Rep-Net model can capture multi-scale and rotationally robust road features and obtain high-yield feature mapping through continuous multi-branch learning of the stacked MHK module. However, the cost of multi-branch learning is the sharp increase in memory usage and parameter quantity, which seriously affects the inference speed of the model. Therefore, the structure reparameterization idea is introduced, and A 2 The method for decoupling the training period and the inference period of the Rep-Net model is to equivalently convert the multi-branch MHK, Stem1 and Stem2 modules into a straight-laid single-branch structure after the model training is completed, and to construct an independent inference model. This single-branch inference model after decoupling can reduce the complexity of the model while retaining the strong representation ability obtained during training, and significantly improve the actual inference speed of the model.

[0023] A 2 The reparameterization of the Rep-Net mainly includes two steps: absorption of each batch normalization layer (Batch Normalization, BN) by the convolution layer, and reparameterization of the MHK, Stem1 and Stem2 modules; A 2 The reparameterization of the Rep-Net includes absorption of each batch normalization layer (Batch Normalization, BN) by the convolution layer. The absorption method of the BN layer by the convolution layer is that the MHK, Stem1 and Stem2 modules in the model have BN layers after the convolution layers in the multi-branch structure. Considering that convolution and BN are both linear operations and have additivity, absorption fusion is performed. The operations of the convolution layer and the BN layer are as follows:

[0024] y conv = ω · x + b (5)

[0025]

[0026] In formula (5) and formula (6), x represents input, y conv and y BN respectively represent the output of the convolution layer and the BN layer, and ω and b are respectively the weight and the bias of the convolution, and μ, σ, γ and η are respectively the mean value, the variance, the scaling variable and the translation variable of the BN layer, in the training period, ω, b, μ, σ, γ and η are constantly updated, and in the inference period, the above six parameters are fixed values, combining formula (5) and formula (6), a new expression of the convolution layer after absorbing the BN layer can be obtained:

[0027]

[0028]

[0029] In formula (7) and formula (8), y BN(conv) represents the output of the new convolution layer, and respectively represent the weight and the bias of the new convolution layer.

[0030] The above A 2 The Rep-Net reparameterization also includes the reparameterization of the MHK module, the multi-branch part of the MHK module is composed of four parallel convolutions of 3x3, 1x3, 3x1 and 1x1 and one identity connection, the four convolution branches first absorb the BN layer, and then the 1x3, 3x1 and 1x1 convolutions are zero-padded to 3x3 size respectively, that is, they are equivalent to special 3x3 convolution kernels with many zero parameters, for the identity connection branch, the input feature is equal to the output feature, so it is equivalent to a special 1x1 convolution, the weight of the i-th 1x1 convolution in the i-th channel is 1, and the rest are zero, that is, the input is not changed after convolution, therefore, the identity connection branch can also be zero-padded to a 3x3 convolution kernel after absorbing the BN layer, using the additivity of convolution, the equivalent 3x3 convolution kernels of the five branches are added and fused into a new 3x3 convolution, and the weight and the bias are the sum of the original values, thus, the five-branch MHK module is reparameterized into a single-branch structure of 1 3x3 convolution + 1 ReLU activation, that is, the RepMKH module.

[0031] A 2Rep-Net reparameterization also includes the reparameterization of the Stem1 and Stem2 modules. The reparameterization of Stem1 and Stem2 modules is similar to that of the MHK module. For the Stem1 module, the 1×1 convolutional branch is padded with zeros to expand to a 4×4 size after absorbing the BN layer, and then added and merged with the 4×4 convolutional branch after absorbing the BN layer, thus reparameterizing the Stem1 module into a single-branch RepStem1 module. For the Stem2 module, since it needs to be added with the D3×3 convolution, the identity connection branch is equivalent to a 1×1 convolution with all channel weights set to 1. After absorbing the BN layer, it is padded with zeros to convert it into a 3×3 convolutional kernel, and then merged with the D3×3 convolutional branch, thus reparameterizing the Stem2 module into a single-branch RepStem2 module.

[0032] At this point, A 2 The multi-branch modules MHK, Stem1, and Stem2 of the Rep-Net model can all be reparameterized into a linear single-branch structure RepMHK, RepStem1, and RepStem2 during the inference cycle. However, the ACA module, due to its non-linear pooling operations, is difficult to merge with identity connections and therefore cannot be reparameterized. But the GAP, GMP, and F... C×1×1 Performing 1×k convolutions is a lightweight operation, so the complexity is extremely low and it does not affect the inference speed of the model.

[0033] This invention proposes a lightweight network based on structural reparameterization and adaptive attention (A 2 Rep-Net is used as a model for recognizing the type of road surface in front of a vehicle. 2 The Rep-Net model is characterized and innovative in the following three aspects:

[0034] (1) A lightweight and efficient attention mechanism was designed, which can adaptively aggregate spatial context information according to feature size and adaptively perform local cross-channel interaction according to feature dimension. By calibrating the feature response between channels, it can focus on features that are highly correlated with the road surface, thereby improving the model's ability to classify and identify complex road surfaces.

[0035] (2) The concept of structural reparameterization is introduced to decouple the training cycle and inference cycle of the network. During training, multi-branch learning is carried out by constructing horizontal / vertical / square / point-shaped parallel heterogeneous convolution kernels to obtain high-margin feature representation. During inference, the multi-branch structure is equivalently converted into a linear single-branch structure, so as to obtain a lightweight deployment model and significant inference acceleration without reducing (sacrificing) the model performance.

[0036] (3)A 2 The Rep-Net model can effectively identify 7 types of pavements, including asphalt, cement, ice and snow, sand, tiles, stone slabs and wet asphalt (wet and slippery), in a complex environment, and achieves a full-scene classification accuracy of 99.14% and a new-scene classification accuracy of 96.48% with a lightweight parameter quantity of 6.57M, and has a high reasoning speed of 496.28 frames·s -1 , which is beneficial to the actual deployment of the model on the vehicle end.

[0037] To sum up, in the construction method of the present application, the proposed model can effectively identify the pavement type in a complex driving environment, and the model achieves a good balance of accuracy, real-time performance and lightweight, has high adaptability in complex and variable scenes, and has obvious use effect in the pavement type identification task in front of the vehicle. The problem that the existing method cannot balance accuracy and speed and is difficult to deploy on the vehicle end is solved. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 For A 2 The architectural design diagram of the Rep-Net model. Figure 1 In (a), A 2 The 5-stage architecture diagram of Rep-Net, Figure 1 In (b), the detailed structure diagram of the module.

[0039] Figure 2 It is the structure diagram of the ACA module which has self-adaptability in spatial aggregation and channel interaction.

[0040] Figure 3 It is the reparameterization diagram of the MHK module with multi-branch heterogeneous convolution kernel. Figure 3 In (a), it is the multi-branch reparameterization process diagram of the MHK module, Figure 3 In (b), it is the structure diagram of the RepMKH module.

[0041] Figure 4 It is the reparameterization diagram of the Stem1 module and the Stem2 module. Figure 4 In (a), it is the multi-branch reparameterization process diagram of the Stem1 module, Figure 4 In (b), it is the structure diagram of the RepStem1 module, Figure 4 In (c), it is the multi-branch reparameterization process diagram of the Stem2 module, Figure 4 In (d), it is the structure diagram of the RepStem2 module.

[0042] Figure 5 、 Figure 6 、 Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 andFigure 11 Fig. 1 is a sample example diagram of an RTCD data set, which refers to a real driving scene road surface type classification data set. Figure 5 、 Figure 6 、 Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 and Figure 11 The displayed road surface types are asphalt, cement, ice and snow, sand, brick, stone and wet asphalt respectively.

[0043] Figure 12 、 Figure 13 Fig. 4 is a confusion matrix diagram of the Rep-Net model on the data set RTCD1 and the data set RTCD2. 2 Fig. 4 is a confusion matrix diagram of the Rep-Net model on the data set RTCD1 and the data set RTCD2. Figure 12 Fig. 5 is a diagram of experimental results of the data set RTCD1, Figure 13 Fig. 6 is a diagram of experimental results of the data set RTCD2. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by a person of ordinary skill in the art without making creative efforts based on the embodiments in the present application are within the scope of protection of the present application.

[0045] Embodiment 1: As shown in Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 、 Figure 11 、 Figure 12 、 Figure 13 As shown in the construction method of the complex road surface rapid identification model, the construction method of the complex road surface rapid identification model comprises the following process steps:

[0046] ① Construct a feature extraction backbone network with horizontal, vertical, square and point-shaped multi-branch heterogeneous convolution as the core, that is, construct a multi-branch heterogeneous convolution kernel MHK (to ensure sufficient extraction of road surface features) for multi-branch learning;

[0047] ②Adaptive Channel Attention (ACA) is designed to focus on high-correlation features, and to adaptively aggregate spatial context information according to feature size and to adaptively perform local cross-channel interaction according to feature dimension. By calibrating the feature responses between channels, the model focuses on features with high correlation to the road surface.

[0048] ③Structural Reparameterization is introduced to decouple the training and inference periods of the model (network). During training, high-dimensional feature representation is obtained through the above multi-branch learning, while during inference, the multi-branch structure is equivalent to a straight-laid single-branch structure. Without reducing the performance of the model, a lightweight deployment model and inference speed are obtained. That is, the structural Reparameterization strategy (Rep) is introduced to equivalently convert the multi-branch MHK module into a single-branch RepMHK module to obtain a lightweight and efficient model inference. On this basis, the A 2 Rep-Net model is proposed, which achieves a good trade-off between accuracy and speed in the road type identification task.

[0049] It needs to be explained that the real driving scene contains complex backgrounds and foregrounds such as sky, vehicles, pedestrians, and buildings, which will introduce a large amount of redundant information to interfere with the identification of the road surface. Therefore, the model is required to have high road feature representation ability and information focusing ability. At the same time, the model needs to be lightweight and efficient to facilitate deployment on a car-grade chip. In view of this, the Multi-branch Heterogeneous Kernel (MHK) is constructed to ensure sufficient extraction of road features, the Adaptive Channel Attention (ACA) module is designed to focus on high-correlation features, and the structural Reparameterization strategy is introduced to equivalently convert the multi-branch MHK module into a single-branch RepMHK module to achieve lightweight and efficient model inference. On this basis, the A 2 Rep-Net model is proposed, which achieves a good trade-off between accuracy and speed in the road type identification task.

[0050] As Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 11The road surface types shown are complex road surfaces such as asphalt road surface, cement road surface, ice and snow road surface, sandy soil road surface, tile road surface, stone slab road surface, and wet asphalt (slippery) road surface, etc. For A 2 The Rep-Net overall architecture (training cycle) is shown in the following table: 2 The Rep-Net model is a 5-stage convolutional neural network architecture, as shown in the following table: Figure 1 As shown in the table, the three parameters in the module are the channel number, height, and width of its input feature map, respectively; the 0th stage is the input stage, which is composed of 1 Stem1 module and 1 ACA module, wherein the Stem1 module is composed of 1 4x4 convolution branch and 1 1x1 convolution branch, both with a step size of 2, as shown in the following table: Figure 1 As shown in the table, the Stem1 module is composed of 1 4x4 convolution branch and 1 1x1 convolution branch, both with a step size of 2, which serves to downsample the input image and expand the channel dimension, reducing the inherent redundancy of natural images while preliminarily extracting features; the ACA module uses an attention mechanism to adaptively focus on the features at the channel level, and the specific structure is described below. The 1st to 4th stages are the main feature extraction stages, each composed of 1 Stem2 module, several MHK modules, and 1 ACA module, as shown in the following table: Figure 1 As shown in the table, the Stem2 module is the input module of each stage, which first uses a 1x1 convolution with a step size of 2 to downsample and expand the channel dimension, and then uses a lightweight 3x3 deep convolution (denoted as D3x3) to extract road surface features, and a constant connection branch is set to avoid gradient disappearance problem, the MHK module is the main feature extraction module of each stage, which is composed of a square 3x3, a horizontal 1x3, a vertical 3x1, a point 1x1, and a constant connection branch in parallel, the multiple convolution kernels of different sizes can make the model have a multi-scale receptive field, and the asymmetric convolution kernel can make the model have robustness to rotation distortion, which overall improves the road surface feature representation ability of the model. The number of MHK modules in the 1st to 4th stages is 2, 4, Θ, and 1, respectively, by changing the number of MHK stacking Θ in the 3rd stage, the depth of the model is adjusted; at the end of the 1st to 4th stages, the ACA module is also embedded to adaptively focus on the features, as the network deepens, the feature map size continuously decreases, in order to ensure that the model can fully learn the road surface features in a small scale space, a channel scaling factor Φ is set in the 4th stage to increase the feature dimension, and a high-dimensional feature representation is obtained, after the 4th stage, the network uses a layer of full connection and Softmax function to output the prediction result of the road surface type.

[0051] As shown in the table, the Stem1 module is composed of 1 4x4 convolution branch and 1 1x1 convolution branch, both with a step size of 2, which serves to downsample the input image and expand the channel dimension, reducing the inherent redundancy of natural images while preliminarily extracting features; the ACA module uses an attention mechanism to adaptively focus on the features at the channel level, and the specific structure is described below. The 1st to 4th stages are the main feature extraction stages, each composed of 1 Stem2 module, several MHK modules, and 1 ACA module, as shown in the following table: Figure 1 , Figure 2 , Figure 3 , Figure 4As shown, for the ACA module (training period), in order to optimize the output feature map of each stage of the model and make it more focused on the features with high correlation with the road surface, the ACA module has self-adaptability in spatial aggregation and channel interaction, such as Figure 2 As shown, the ACA module is a self-adaptive spatial aggregation module, and for the input feature map F C×H×W , C is the number of channels, H and W are the height and width of the feature map respectively, and the ACA module first aggregates the spatial context information using two global pooling branches: one is to compress the feature map through global average pooling (GAP) to retain the average response feature F A C×1×1 , that is, the global overall information of the spatial context; the other is to compress the feature map through global maximum pooling (GMP) to retain the maximum response feature F M C×1×1 , that is, to emphasize the local prominent details in the spatial context, then the ACA module fuses the outputs of the two global pooling branches; considering that the local details contained in the feature map are more abundant when the size of the feature map is large, in order to fully capture the detailed information in the context during spatial aggregation, the GMP branch is set with a weight coefficient β, and the fusion weight of the GMP branch is adaptively adjusted according to the feature map size [H, W], that is, there is a mapping f(β) between [H, W] and β, if a linear mapping is used, that is, f = m × β - n, then the expression capacity is limited, in order to realize the adaptive fusion of nonlinearity, and considering that the size of the feature map is mostly a power of 2, the average value of H and W is taken as the feature size, and:

[0052]

[0053] Therefore, for a given feature map size [H, W], the proportion of the GMP branch is adaptively obtained as follows:

[0054]

[0055] In formula (1) and formula (2), m and n are parameters, and m = 40 and n = 1 are taken, through the nonlinear mapping, the large size feature map will obtain a higher GMP branch weight, it should be noted that the value of β is always less than 1, which means that the proportion of the GAP branch in the ACA module is always higher than that of the GMP branch, that is, the global overall information of the feature map always dominates when aggregating the spatial context. The above ACA module is a self-adaptive channel interaction module, and after self-adaptive spatial aggregation, the feature map F C×1×1 is obtained A C×1×1 + β × F M C×1×1, the ACA module further performs adaptive channel calibration on the aggregated feature F C×1×1 , focusing on the features related to the road surface height, which cancels the channel dimension reduction operation to obtain more effective channel attention, while using a one-dimensional convolution 1xk of size k to quickly obtain local cross-channel interaction, where the parameter k is adaptively obtained from the input channel number C, considering that the channel number is also a power of 2, then let:

[0056] C=g(k)=2 p×k-q (3)

[0057] Therefore, for a given channel number C, the value of k is adaptively obtained as follows:

[0058]

[0059] In equation (3), equation (4), p and q are parameters, respectively taking 2 and 1; considering that the convolution kernel size is mostly odd, here we use |■| odd The nearest odd number is taken upwards, and the feature map F C×1×1 After adjusting the channel weight by 1xk convolution, the normalized channel attention map A C×1×1 is generated through the SiLU activation function. C×H×W The original input feature map F C×1×1 is multiplied with the channel attention map A C×H×W to obtain the feature map F' recalibrated by the ACA module. 2 The ACA module is embedded at the end of each stage of the network to perform adaptive channel calibration on the feature map, strengthen the features related to the road surface type height, suppress redundant features with low relevance, and improve the model's classification and discrimination ability for complex road surfaces.

[0060] As shown in Figure 1 , Figure 2 , Figure 3 , Figure 4 , for structure reparameterization (inference period), in the training period, A 2 The Rep-Net model can capture multi-scale and rotationally robust road features and obtain high-dimensional feature mapping through the continuous multi-branch learning of the stacked MHK modules. However, the cost of multi-branch learning is the rapid increase of memory usage and parameter quantity, which seriously affects the inference speed of the model. Therefore, the structure reparameterization idea is introduced. 2 The method of decoupling the training period and the inference period of the Rep-Net model is to convert the multi-branch MHK, Stem1 and Stem2 modules into equivalent straight-laid single-branch structures after the model training is completed, and to construct an independent inference model. This decoupled single-branch inference model can reduce the complexity of the model while retaining the strong representation ability obtained during training, significantly improving the actual inference speed of the model.

[0061] A 2 Rep-Net's reparameterization mainly includes two steps: the absorption of each batch normalization (BN) layer (BN layer) by the convolutional layers, and the reparameterization of the MHK, Stem1, and Stem2 modules; A 2 Rep-Net's reparameterization includes the absorption of each batch normalization (BN) layer by the convolutional layers. The method for absorbing the BN layer from the convolutional layers is that in the MHK, Stem1, and Stem2 modules of the model, each convolutional layer in its multi-branch structure is followed by a BN layer. Considering that both convolution and BN are linear operations and have additive properties, they are absorbed and fused. The operations of the convolutional layers and BN layers are as follows:

[0062] yconv=ω·x+b (5)

[0063]

[0064] In equations (5) and (6): x represents the input, y conv and y BN ω and b represent the outputs of the convolutional layer and the BN layer, respectively. ω and b are the weights and biases of the convolution, respectively. μ, σ, γ, and η are the mean, variance, scaling variable, and translation variable of the BN layer, respectively. During the training period, ω, b, μ, σ, γ, and η are continuously updated, while during the inference period, all six parameters are fixed values. Combining formulas (5) and (6), we can obtain the new expression after the convolutional layer absorbs the BN layer:

[0065]

[0066]

[0067] In equations (7) and (8): y BN(conv) Represents the output of the new convolutional layer. and These represent the weights and biases of the new convolutional layer, respectively.

[0068] The above A 2 Rep-Net's reparameterization also includes the reparameterization of the MHK module. The multi-branch part of the MHK module consists of four parallel convolutions (3×3, 1×3, 3×1, and 1×1) and one identity connection, such as... Figure 3As shown, the four convolutional branches first absorb the BN layer, and then pad the 1×3, 3×1, and 1×1 convolutions with zeros to expand them to a 3×3 size, effectively transforming them into special 3×3 convolutional kernels with many zero parameters. For the identity connection branch, the input features are always equal to the output features, so it is equivalent to a set of special 1×1 convolutions: the weight of the i-th channel of the i-th 1×1 convolution is 1, and the rest are zero, meaning that the input is not changed after convolution. Therefore, the identity connection branch can also absorb the BN layer and pad it with zeros to transform it into a 3×3 convolutional kernel. Utilizing the additivity of convolution, the equivalent 3×3 convolutional kernels of the five branches are added and merged into a new 3×3 convolution, whose weights and biases are the sum of the original values. Thus, the 5-branch MHK module is reparameterized into a single-branch structure of one 3×3 convolution + one ReLU activation, i.e., the RepMKH module. The above A... 2 Rep-Net reparameterization also includes the reparameterization of the Stem1 and Stem2 modules, which is similar to that of the MHK modules. Figure 4 As shown. For the Stem1 module, the 1×1 convolutional branch is padded with zeros to expand to a 4×4 size after absorbing the BN layer, and then added and merged with the 4×4 convolutional branch after absorbing the BN layer, thus reparameterizing the Stem1 module into a single-branch RepStem1 module; for the Stem2 module, since it needs to be added with the D3×3 convolution, the identity connection branch is equivalent to a 1×1 convolution with all channel weights set to 1, which is padded with zeros after absorbing the BN layer to convert it into a 3×3 convolutional kernel, and then merged with the D3×3 convolutional branch, thus reparameterizing the Stem2 module into a single-branch RepStem2 module.

[0069] At this point, A 2 The multi-branch modules MHK, Stem1, and Stem2 of the Rep-Net model can all be reparameterized into a linear single-branch structure RepMHK, RepStem1, and RepStem2 during the inference cycle. However, the ACA module, due to its non-linear pooling operations, is difficult to merge with identity connections and therefore cannot be reparameterized. But the GAP, GMP, and F... C×1×1 Performing 1×k convolutions is a lightweight operation, so the complexity is extremely low and it does not affect the inference speed of the model.

[0070] The following is the experimental section (experimental setup) of this invention:

[0071] 1.1 Database.

[0072] Considering the impact of road surface unevenness and adhesion coefficient on vehicle active control, this study focuses on seven types of road surfaces: asphalt, cement, snow and ice, sand, paved roads, brick roads, and wet asphalt. Over 100 kilometers of real-world driving records captured by onboard cameras were collected. To ensure data diversity, 17,500 road surface images of various driving scenarios under different weather and lighting conditions were selected, with an original resolution of 1920×1080. Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 As shown, after labeling each road surface with its ground truth (GT), a Road Type Classification Dataset (RTCD) for real-world driving scenarios was constructed.

[0073] 1.2 Model Training.

[0074] A 2 The Rep-Net model was trained using an NVIDIA RTX 3090 GPU and PyTorch 1.11.0 framework on Windows 10. The initial learning rate was set to 5e-3, and it was updated by 0.9 times every five iterations, with a maximum of 300 iterations. The optimizer was AdamW, with a weight decay value of 0.01, and cross-entropy loss was used for training supervision. The batch size was set to 128, Shuffle was set to True, and a label smoothing strategy was employed. Based on the RTCD dataset, two training and testing strategies (testing methods) were proposed:

[0075] Training and testing strategy 1, similar to most image classification tasks, randomly divides all samples in the dataset into a training set (80%) and a test set (20%). This strategy can test the model's ability to recognize road surfaces in all scenes. The dataset divided in this way is denoted as RTCD1.

[0076] Training and testing strategy 2: The samples in the training set and the test set are from dashcam videos in different scenarios. This strategy can simulate and test the model's recognition ability when encountering new scenarios during driving. The dataset divided in this way is denoted as RTCD2.

[0077] 1.3 Evaluation Indicators.

[0078] To evaluate A 2 The classification performance of the Rep-Net model is evaluated using the Macro-F1 score and confusion matrix. Macro-F1 is the average F1 score for all road surface types, defined as follows:

[0079]

[0080]

[0081] In formula (9) and formula (10): TP and TN respectively represent the number of correctly predicted positive and negative samples, and FP and FN respectively represent the number of incorrectly predicted positive and negative samples; i = 1, 2, …, N, i represents different road surface types, N is the total number of types, and N = 7. The closer the Macro-F1 score is to 1, the better the overall classification performance of the model on the road surface. At the same time, the parameter quantity, floating point operation quantity (FLOPs) and inference speed in frames per second are introduced to respectively evaluate the size, calculation complexity and efficiency of the model.

[0082] 1.4 Experimental results and analysis.

[0083] A 2 The Rep-Net model achieves Macro-F1 scores of 99.14% (for full-scene recognition) and 96.48% (for new-scene recognition) on RTCD1 and RTCD2, respectively. Figure 12 、 Figure 13 A 2 The confusion matrix of the Rep-Net model on RTCD1 and RTCD2 is shown. It can be seen that on RTCD1 for full-scene recognition, the recognition accuracy of each type of road surface is not less than 97.8%. Among them, the recognition accuracy of ice and snow, cement, sand, stone and tile 5 types of road surfaces reaches more than 99.6%. And on RTCD2 for new-scene recognition, the recognition accuracy of cement, sand, tile, stone 4 types of road surfaces reaches more than 96.8%.

[0084] To further verify the performance of A 2 The performance of the Rep-Net model is compared with seven mainstream classification models, including ResNet, ShuffleNetV2, MobileNetV2, DenseNet, RepVGG, RepGhostNet and ConvNeXt, which are representative models in the same category in recent years, and the results are shown in Table 1. It can be seen that on RTCD1 for full-scene recognition and RTCD2 for new-scene recognition, the Macro-F1 scores [99.14%, 96.48%] of the Rep-Net model are better than those of other models, which are higher than those of the second place ConvNeXt and the third place RepVGG by [0.13%, 1.28%] and [0.16%, 3.28%], respectively. 2 2 ​The accuracy advantage of the Rep-Net model on the RTCD2 is higher than that of the RTCD1, indicating that the proposed model is more suitable for new driving scenes and has better generalization capability. 2 The inference speed of the Rep-Net model is also better, and the average processing speed of 496.28 frames per second is 43% and 52% faster than the MobileNetV2 and the ShuffleNetV2 respectively, and the parameter amount of 6.57M is also relatively light.

[0085] In summary, the A 2 The Rep-Net model relies on the use of adaptive attention mechanism and structural reparameterization strategy, and considers the accuracy, real-time performance and light weight, and good accuracy-speed balance is achieved in the road type identification task, which is beneficial to the deployment of the model on the vehicle side, and has higher adaptability to new scenes.

[0086] Table 1 Comparison of road identification performance of different models

[0087]

[0088] In combination with the above, in the complex driving environment, quickly and accurately identifying the road type in front of the vehicle is a key prerequisite for the vehicle active control system to make a timely prediction. In view of the problem that the existing method cannot consider the accuracy and speed and is difficult to deploy on the vehicle side, a road classification model based on structural reparameterization and adaptive attention is proposed, which can quickly and accurately distinguish complex road surfaces such as asphalt, cement, ice and snow, sandy soil, tile, stone plate and wet asphalt (wet) in front of the vehicle. First, a feature extraction backbone network is constructed, which takes multi-branch heterogeneous convolution such as horizontal / vertical / square / point as the core. Secondly, a light and efficient attention mechanism is proposed, which can adaptively aggregate spatial context information according to the feature size, and adaptively perform local cross-channel interaction according to the feature dimension, so that the model focuses on the high correlation of road features. On this basis, the structural reparameterization idea is introduced to decouple the training period and the inference period of the model. In the training, high-dimensional feature representation is obtained through multi-branch learning, while in the inference, the multi-branch structure is equivalent to a straight-laid single branch structure, which obtains a lightweight deployment model and significant inference acceleration without sacrificing the performance of the model. The experimental results show that the proposed model can effectively identify the road type in complex driving environment, and the parameter amount of 6.57M achieves 99.14% of the full-scene classification accuracy and 96.48% of the new-scene classification accuracy, while the high inference speed of 496.28 frames per second. Compared with other existing mainstream models, the model proposed in the application achieves a good balance of accuracy, real-time performance and light weight, has high adaptability to complex and variable scenes, and has obvious effect in the road type identification task in front of the vehicle.

Claims

1. A method for constructing a fast road surface recognition model, which uses a lightweight network with reparameterized structure and adaptive attention as a model for recognizing the road surface type in front of a vehicle, and distinguishes the road surface in front of the vehicle, characterized in that... The method for constructing the rapid identification model for complex road surfaces includes the following process steps: ① Construct a feature extraction backbone network with horizontal, vertical, square, and dot-shaped multi-branch heterogeneous convolutions as the core, that is, construct a multi-branch heterogeneous convolution kernel (MHK) and perform multi-branch learning; ② A lightweight and efficient attention mechanism is adopted, and an adaptive channel attention module (ACA) is designed to focus on highly relevant features. It adaptively aggregates spatial context information according to feature size and adaptively performs local cross-channel interaction according to feature dimension. By calibrating the feature response between channels, the model focuses on features that are highly relevant to the road surface. ③ A structural reparameterization method is introduced to decouple the model's training and inference cycles. During training, high-margin feature representations are obtained through multi-branch learning, while during inference, the multi-branch structure is equivalently converted to a linear single-branch structure. This achieves lightweight model deployment and inference speed without compromising model performance. Specifically, the structural reparameterization strategy converts the multi-branch MHK module into a single-branch RepMHK module to obtain lightweight and efficient model inference. Based on this, an A / B algorithm is adopted to achieve a trade-off between accuracy and speed in road type recognition tasks. 2 Rep-Net model; The above A 2 The Rep-Net model is a 5-stage convolutional neural network architecture, with the three parameters within each module being the number of channels, height, and width of its input feature map, respectively. Stage 0 is the input stage, consisting of one Stem1 module and one ACA module. The Stem1 module consists of one 4×4 convolutional branch and one 1×1 convolutional branch, both with a stride of 2. Stages 1 to 4 are the main feature extraction stages, each consisting of one Stem2 module, several MHK modules, and one ACA module. The Stem2 module is the input module for each stage. It first uses a 1×1 convolution with a stride of 2 for downsampling and expands the channel dimension, and then uses a lightweight 3×3 depthwise convolution to extract road surface features. The 3×3 depthwise convolution is denoted as D3×3. At the same time, an identity connection branch is set to avoid the gradient vanishing problem. The MHK module is the main feature extraction module for each stage, consisting of four heterogeneous convolutional branches in parallel: a square 3×3, a horizontal 1×3, a vertical 3×1, and a point 1×1, and an identity connection branch. A 2 Rep-Net's reparameterization also includes the reparameterization of the MHK module. The multi-branch part of the MHK module consists of four parallel convolutions (3×3, 1×3, 3×1, and 1×1) and one identity connection. The four convolutional branches first absorb the BN layer, and then the 1×3, 3×1, and 1×1 convolutions are padded with zeros to expand to a 3×3 size, which is equivalent to transforming them into special 3×3 convolutional kernels with many zero parameters. For the identity connection branch, the input features are always equal to the output features, so it is equivalent to a set of special 1×1 convolutions. Product: The weight of the i-th channel of the i-th 1×1 convolution is 1, and the rest are zero, which means that the input is not changed after the convolution. The identity connection branch can also absorb the BN layer and fill zeros to convert it into a 3×3 convolution kernel. Taking advantage of the additivity of convolution, the equivalent 3×3 convolution kernels of the 5 branches are added and merged into a new 3×3 convolution, whose weights and biases are the sum of the original values. Thus, the 5-branch MHK module is reparameterized into a single-branch structure of 1 3×3 convolution + 1 ReLU activation, namely the RepMKH module.

2. The method for constructing a rapid identification model for complex road surfaces according to claim 1, characterized in that... The road surface types are asphalt road surface, cement road surface, snow and ice road surface, sand road surface, paved road surface, stone slab road surface and wet asphalt road surface.

3. The method for constructing a rapid identification model for complex road surfaces according to claim 1, characterized in that... The number configurations of the MHK modules in stages 1 to 4 are 2, 4, Θ, and 1, respectively. The depth of the model is adjusted by changing the number of MHK stacks Θ in stage 3. At the end of stages 1 to 4, an ACA module is also embedded to adaptively focus on the features. As the network deepens, the feature map size decreases continuously. In order to ensure that the model can fully learn the road features in a small-scale space, a channel scaling factor Φ is set in stage 4 to increase the feature dimension and obtain a high-margin feature representation. After stage 4, the network uses a fully connected layer and a Softmax function to output the prediction results of the road type.

4. The method for constructing a rapid identification model for complex road surfaces according to claim 1, characterized in that... For the input feature map F C×H×W C represents the number of channels, and H and W represent the height and width of the feature map, respectively. The ACA module first uses two global pooling branches to aggregate spatial context information: one is to compress the feature map through global average pooling (GAP), retaining the average response feature F of each dimension. A C×1×1 This refers to the global and overall information that reflects the spatial context; Another approach is to compress the feature map using Global Max Pooling (GMP) to retain the maximum response feature F in each dimension. M C×1×1 This emphasizes local details within the spatial context. Subsequently, the ACA module fuses the outputs of the two global pooling branches. A weight coefficient β is set for the GMP branch, and the fusion weights of the GMP branch are adaptively adjusted based on the feature map size [H,W]. There is a mapping f(β) between [H,W] and β. If a linear mapping is used, i.e., f=m×β-n, the expressive power is limited. To achieve non-linear adaptive fusion, and given that the feature map size is a power of 2, the average of H and W is used as the feature size. Let: Therefore, for a given feature map size [H, W], the weight of the GMP branch is adaptively obtained by the following formula: In equations (1) and (2), m and n are parameters. Take m = 40 and n = 1. Through this nonlinear mapping, large-size feature maps will obtain higher GMP branch weights.

5. The method for constructing a rapid identification model for complex road surfaces according to claim 4, characterized in that... After adaptive spatial aggregation, the feature map F is obtained. C×1×1 =F A C×1×1 +β×F M C×1×1 The ACA module further refines the aggregation feature F. C×1×1 Adaptive channel calibration is performed to focus on features highly correlated with road surface height. This part cancels channel dimensionality reduction to obtain more effective channel attention. Simultaneously, a one-dimensional convolution of size k (1×k) is used to quickly capture local cross-channel interactions, where the parameter k is adaptively obtained from the number of input channels C. Considering that the number of channels is also a power of 2, let: C=g(k)=2 p×k-q (3) Therefore, for a given number of channels C, the value of k is adaptively obtained by the following formula: In equations (3) and (4): p and q are parameters, taking values ​​of 2 and 1 respectively; considering that the kernel size is mostly odd, |■| is used here. odd Take the nearest odd number upwards, feature map F C×1×1 After adjusting the channel weights through 1×k convolution, a normalized channel attention map A is generated using the SiLU activation function. C×1×1 The original input feature map F C×H×W With channel attention map A C×1×1 The dot product fusion is performed to obtain the feature map F' recalibrated by the ACA module. C×H×W .

6. The method for constructing a rapid identification model for complex road surfaces according to claim 4, characterized in that... For A 2 The method to decouple the training cycle and inference cycle of the Rep-Net model is to convert the multi-branch MHK, Stem1 and Stem2 modules into a linear single-branch structure after the model training is completed, and build an independent inference model. A 2 Rep-Net's reparameterization includes the absorption of batch normalization (BN) layers by convolutional layers. The method for absorbing BN layers from convolutional layers is that in the MHK, Stem1, and Stem2 modules of the model, each convolutional layer in its multi-branch structure is followed by a BN layer. Considering that both convolution and BN are linear operations and additive, they are absorbed and fused. The operations of the convolutional layers and BN layers are as follows: y conv =ω·x+b (5) In equations (5) and (6): x represents the input, y conv and y BN ω and b represent the outputs of the convolutional layer and the BN layer, respectively. ω and b are the weights and biases of the convolution, respectively. μ, σ, γ, and η are the mean, variance, scaling variable, and translation variable of the BN layer, respectively. During the training period, ω, b, μ, σ, γ, and η are continuously updated, while during the inference period, all six parameters are fixed values. Combining formulas (5) and (6), we can obtain the new expression after the convolutional layer absorbs the BN layer: In equations (7) and (8): y BN(conv) Represents the output of the new convolutional layer. and These represent the weights and biases of the new convolutional layer, respectively.

7. The method for constructing a rapid identification model for complex road surfaces according to claim 4, characterized in that... A 2 Rep-Net's reparameterization also includes the reparameterization of the Stem1 and Stem2 modules. For the Stem1 module, the 1×1 convolutional branch is padded with zeros to expand to a 4×4 size after absorbing the BN layer, and then added and merged with the 4×4 convolutional branch after absorbing the BN layer, thus reparameterizing the Stem1 module into a single-branch RepStem1 module. For the Stem2 module, since it needs to be added with a D3×3 convolution, the identity connection branch is equivalent to a 1×1 convolution with all channel weights set to 1. After absorbing the BN layer, it is padded with zeros to convert it into a 3×3 convolutional kernel, and then merged with the D3×3 convolutional branch, thus reparameterizing the Stem2 module into a single-branch RepStem2 module.

Images