Road surface recognition method, computer-readable storage medium, controller, and vehicle

Abstract

The application discloses a road surface identification method, a computer readable storage medium, a controller and a vehicle, and relates to the technical field of automatic driving. The road surface identification method comprises the following steps: acquiring to-be-identified image data; segmenting road surface information in the to-be-identified image data by using a semantic segmentation model to obtain segmented image data; and identifying the segmented image data by using a deep learning model to obtain a road surface type. The road surface identification method can identify the road surface type of a current scene in real time by segmenting to-be-identified image data by using a semantic segmentation model and then identifying the segmented image data by using a deep learning model, and provides an important decision basis for automatic driving. Meanwhile, the road surface type is identified in two stages, and the accuracy of road surface identification is improved.

Description

Road surface recognition method, computer-readable storage medium, controller and vehicle Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a road surface recognition method, a road surface recognition device, a computer program product, a computer-readable storage medium, a controller, and a vehicle. Background Technology

[0002] Automatic road surface recognition is a crucial function in autonomous driving environmental perception systems. It uses an onboard monocular camera to automatically identify various road surfaces and transmits the identified information via a bus. However, existing road surface recognition methods suffer from poor recognition accuracy and insufficient training data. Summary of the Invention

[0003] The present invention is proposed to address at least one of the aforementioned problems. According to a first aspect of this application, a road surface recognition method is provided, the method comprising: acquiring image data to be recognized; segmenting road surface information in the image data to be recognized using a semantic segmentation model to obtain segmented image data; and recognizing the segmented image data using a deep learning model to obtain the road surface type.

[0004] In one embodiment of this application, the semantic segmentation model includes the Leaky ReLU activation function.

[0005] In one embodiment of this application, the semantic segmentation model segments road surface information in the image data to be identified, including: inputting the image data to be identified into the semantic segmentation model to segment out a region of interest including a road surface area and a region of non-interest including a non-road surface area; setting the pixel value of the pixel in the region of non-interest to a preset value to obtain the segmented image data including the region of interest and the region of non-interest.

[0006] In one embodiment of this application, the deep learning model includes: ACFNet model, Deeplabv3 model, and SegNet model.

[0007] In one embodiment of this application, the deep learning model further includes a weight allocation unit, which is connected to the ACFNet model, the Deeplabv3 model, and the SegNet model, respectively.

[0008] In one embodiment of this application, an improved ResNet network is used as the backbone network of the ACFNet model, an improved RepVGG network is used as the backbone network of the Deeplabv3 model, and an improved RegNet network is used as the backbone network of the SegNet model.

[0009] In one embodiment of this application, the improved ResNet network includes a Coordinate attention mechanism; the improved RepVGG network includes a CBAM attention mechanism; and the improved RegNet network includes an FcaNet attention mechanism.

[0010] In one embodiment of this application, after recognizing the segmented image data using a deep learning model, the method further includes: using a dynamics module to obtain a second road surface type for the region corresponding to the image data to be recognized; and obtaining the road surface type based on the second road surface type and the first road surface type and its proportion identified by the deep learning model.

[0011] In one embodiment of this application, the process of the deep learning model recognizing the segmented image data includes: dividing the segmented image data into multiple sub-regions; recognizing the sub-regions to obtain the first road surface type and its proportion corresponding to each sub-region.

[0012] In one embodiment of this application, obtaining the road surface type based on the second road surface type and the first road surface type and its proportion identified by the deep learning model includes: obtaining the second road surface type corresponding to the target sub-region, and the second road surface type of the target sub-region identified by the dynamics module at the previous K time steps; obtaining the first road surface type and its proportion corresponding to the target sub-region, and the first road surface type and its proportion of the next sub-region corresponding to the target sub-region; and obtaining the road surface type based on the obtained multiple second road surface types and multiple first road surface types.

[0013] In one embodiment of this application, obtaining the road surface type based on the obtained plurality of second road surface types and plurality of first road surface types includes: weighting the proportions of the plurality of first road surface types and weighting the plurality of second road surface types; and obtaining the road surface type based on the results of the two weightings.

[0014] In one embodiment of this application, the semantic segmentation model is obtained by: acquiring first image data, the first image data including road surface regions and non-road surface regions; labeling the road surface and non-road surface regions in the first image data to obtain a first training set; and training the semantic segmentation model using the first training set to obtain a trained semantic segmentation model.

[0015] In one embodiment of this application, after acquiring the first image data, the training process of the semantic segmentation model further includes: deleting the first image data with repeated road surface information, and / or performing style transfer processing on the first image data to increase the number of the first image data.

[0016] In one embodiment of this application, during the training of the semantic segmentation model using the first training set, data augmentation is performed on the samples in the first training set in at least one of the following ways: rotation, translation transformation, shearing transformation, flipping, noise injection, color transformation, dithering, blurring, and category mixing.

[0017] In one embodiment of this application, the method for obtaining the deep learning model includes: acquiring second image data, the second image data being obtained from the recognition device itself; acquiring third image data, the third image data being obtained from the network; annotating the road surface in the second image data and the third image data to obtain a second training set; and training the deep learning model using the second training set to obtain a trained deep learning model.

[0018] In one embodiment of this application, the second image data and the third image data include different types of road surfaces. The different types of road surfaces in the second image data and the third image data are labeled to obtain a second training set.

[0019] In one embodiment of this application, training the deep learning model using the second training set includes: training the ACFNet model, the Deeplabv3 model, and the SegNet model using the second training set respectively; controlling the weight allocation unit to weight the outputs of the ACFNet model, the Deeplabv3 model, and the SegNet model, and adjusting the weight values ​​of the ACFNet model, the Deeplabv3 model, and the SegNet model using a particle swarm optimization algorithm.

[0020] In one embodiment of this application, before annotating the road surface in the second image data and the third image data, the method further includes: preprocessing the second image data and the third image data; wherein, the preprocessing is to perform data stitching processing on the second image data and the third image data to increase the number of the second image data and the third image data.

[0021] According to a second aspect of this application, a road surface recognition device is provided, the device comprising: a data acquisition module for acquiring image data to be recognized; a first data processing module for segmenting road surface information in the image data to be recognized to obtain segmented image data; and a second data processing module for recognizing the segmented image data to obtain a first road surface type.

[0022] According to a third aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements any of the above-described road surface recognition methods.

[0023] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements any of the above-described road surface recognition methods.

[0024] According to a fifth aspect of this application, a controller is provided, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, it implements any of the above-described road surface recognition methods.

[0025] According to a sixth aspect of this application, a vehicle is provided, including: the aforementioned controller.

[0026] According to the road surface recognition method, device, computer program product, computer-readable storage medium, controller, and vehicle provided in the embodiments of this application, the road surface recognition method of this application segments the image data to be recognized through a semantic segmentation model, and then recognizes the segmented image data through a deep learning model. It can identify the road surface type of the current scene in real time, providing an important decision basis for autonomous driving. At the same time, by recognizing the road surface type in two stages, the accuracy of road surface recognition is improved. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 is a schematic flowchart of a road surface recognition method provided in an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of the structure of a deep learning model provided in an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of sub-region segmentation provided in an embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of the road surface recognition device provided in an embodiment of the present invention;

[0032] Figure 5 is a schematic diagram of the controller provided in an embodiment of the present invention;

[0033] Figure 6 is a structural schematic diagram of a vehicle provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0035] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0036] It should be understood that the invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0037] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0038] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0039] The first aspect of this application provides a road surface recognition method, as shown in Figure 1, which includes:

[0040] S1, acquire the image data to be recognized.

[0041] It should be noted that the image data to be identified in the current scene can be acquired through an onboard camera. The onboard camera acts as an image sensor, capturing real-time images of the road surface in front of the vehicle at a frequency of more than 10 frames per second. The real-time images acquired by the onboard camera are in YUV format, and are then converted into three-channel RGB images.

[0042] As an example, the vehicle-mounted camera could be a monocular camera.

[0043] S2 uses a semantic segmentation model to segment road information in the image data to be identified, thereby obtaining segmented image data.

[0044] It should be noted that semantic segmentation is a pixel-level classification task that can separate different information in an image.

[0045] S3 uses a deep learning model to identify segmented image data and obtain road surface types.

[0046] As an example, road surface types can be categorized as mud, sand, snow, grass, and ordinary roads.

[0047] It should be noted that the road surface type recognition results can be applied to assisted driving and can be displayed on dedicated real-world devices for manual verification when needed.

[0048] The road surface recognition method of this invention segments the image data to be recognized using a semantic segmentation model, and then recognizes the segmented image data using a deep learning model. This enables real-time identification of the road surface type in the current scene, providing an important decision-making basis for autonomous driving. At the same time, the two-stage road surface type recognition improves the accuracy of road surface recognition.

[0049] In some embodiments, the semantic segmentation model includes the Leaky ReLU activation function.

[0050] Specifically, the semantic segmentation model is an improved UNet (U-shaped) network; the improved UNet network includes an encoding part, which includes dilated convolutional layers, striped pooling layers, and activation layers. The activation layers sample the Leaky ReLU (LeakyRectified Linear Unit) activation function.

[0051] As an example, the 3×3 convolutions in the UNet network can be replaced with dilated convolutions with a kernel of 5 and a dilation rate of 2, the max pooling layers can be replaced with striped pooling layers, and the ReLU activation function can be replaced with the Leaky ReLU activation function to obtain an improved UNet network.

[0052] It should be noted that after replacing the 3×3 convolutions in the UNet network with dilated convolutions, the input and output dimensions of the model need to be adjusted to ensure that the feature map size after the dilated convolution operation is consistent with the original UNet network. Similarly, after replacing the max pooling layer with a striped pooling layer, the input and output dimensions of each layer after the striped pooling layer need to be adjusted to fit the feature map after striped pooling.

[0053] In this embodiment, dilated convolutional layers enable the semantic segmentation model to have a larger receptive field. Striped pooling is a long, strip-shaped pooling kernel deployed along one spatial dimension. By using striped pooling layers, long-distance relationships in isolated regions can be captured, while maintaining a narrow kernel shape in other spatial dimensions, facilitating the capture of local context and preventing irrelevant regions from interfering with label prediction. During backpropagation, gradients can also be calculated for the parts of the Leaky ReLU activation function input that are less than zero (instead of the ReLU (Rectified Linear Unit) activation function where the output is always 0 when the input is less than zero), thus avoiding the jagged gradient problem.

[0054] In some embodiments, the semantic segmentation model segments road surface information in the image data to be identified, including:

[0055] S211, the image data to be identified is input into the semantic segmentation model to segment the region of interest, which includes the road surface area, and the region of non-interest, which includes the non-road surface area.

[0056] As an example, a semantic segmentation model can first obtain initial segmented image data, where each pixel in the initial segmented image data corresponds to a label, including road surface and non-road surface. Pixels labeled as road surface in the initial segmented image data are grouped into ROI (Region of Interest), and pixels labeled as non-road surface are grouped into non-ROI regions.

[0057] S212, set the pixel values ​​of the pixels in the non-interest region to preset values ​​to obtain segmented image data including the interest region and the non-interest region.

[0058] As an example, the default value can be 0, so that the segmented image data includes road surface information areas and black areas.

[0059] In this embodiment, a semantic segmentation model is used to segment the image data to be identified, ensuring that the image data contains only road surface information. This eliminates the need for manual processing and is suitable for processing large-scale image data. Simultaneously, it reduces interference from deep learning models in road surface type identification, improving accuracy. This avoids subsequent misidentification of surrounding green trees as grass or white truck beds as snow during road surface type recognition.

[0060] In some embodiments, as shown in Figure 2, the deep learning models include: ACFNet (Adaptive Unimodal Cost Volume Filtering Network) model, Deeplabv3 model, and SegNet (Segmentation Network) model.

[0061] Specifically, the deep learning model also includes a weight allocation unit, which is connected to the ACFNet model, the Deeplabv3 model, and the SegNet model, respectively.

[0062] It should be noted that the ACFNet, Deeplabv3, and SegNet models are deep learning semantic segmentation models that demonstrate good classification performance on the ImageNet dataset. The weight allocation unit can dynamically adjust the weights of these models based on their performance and the characteristics of the data. By allocating weights appropriately, the strengths of each model can be fully utilized, improving the overall performance of the model.

[0063] More specifically, an improved ResNet (Residual Network) is used as the backbone network of the ACFNet model, an improved RepVGG (Re-parameterization VGG Network) is used as the backbone network of the Deeplabv3 model, and an improved RegNet (Regularized Network) is used as the backbone network of the SegNet model.

[0064] It should be noted that ResNet, RepVGG, and RegNet are deep learning image classification networks.

[0065] More specifically, the improved ResNet network includes a Coordinate attention mechanism; the improved RepVGG network includes a CBAM (Convolutional Block Attention Module) attention mechanism; and the improved RegNet network includes an FcaNet (Frequency Channel Attention Networks) attention mechanism.

[0066] As an example, the improved ResNet network is specifically: the Coordinate attention mechanism is added to the ResNet network; the improved RepVGG network is specifically: the CBAM attention mechanism is added to the RepVGG network; and the improved RegNet network is specifically: the FcaNet attention mechanism is added to the RegNet network.

[0067] As another example, the Coordinate attention mechanism can be added between the third and fourth convolutional layers in a ResNet network. The CBAM attention mechanism can be added in the output stage of a RepVGG network. The FcaNet attention mechanism can be added in various stages of a RegNet network.

[0068] In this embodiment, by adding a Coordinate attention mechanism to the ResNet network, the expressive power of features can be enhanced by utilizing positional information, thereby improving segmentation accuracy. By adding a CBAM attention mechanism to the RepVGG network, the overall performance of the deep learning model can be further improved. By adding an FcaNet attention mechanism to the RegNet network, the deep learning model's focus on key information can be enhanced, improving segmentation results.

[0069] In some embodiments, after recognizing the segmented image data using a deep learning model, the method further includes:

[0070] S41, use the dynamics module to obtain the second road surface type of the area corresponding to the image data to be identified.

[0071] It should be noted that the dynamics module can be a component on the vehicle that can detect and analyze road conditions in real time. The vehicle's dynamics module can identify the type of road surface beneath the wheels based on the dynamic effects of the contact between the wheels and the road surface. However, actual test results show that this identification result may have some errors.

[0072] As an example, the dynamics module infers the current road surface type based on the vehicle's dynamic characteristics, such as acceleration, velocity, and the coefficient of friction between the tires and the road surface, combined with prior knowledge. These dynamic characteristics can be acquired in real time using onboard sensors, such as acceleration sensors, velocity sensors, and tire pressure sensors.

[0073] S42, based on the second road surface type and the first road surface type identified by the deep learning model and its proportion, the road surface type is obtained.

[0074] In this embodiment, by adding a dynamics module with real-time sensing capabilities, the accuracy and reliability of road surface type identification are improved through comprehensive judgment of road surface type.

[0075] In some embodiments, the process of a deep learning model recognizing segmented image data includes:

[0076] S4211 divides the segmented image data into multiple sub-regions.

[0077] As shown in Figure 3, the segmented image data can be divided into four sub-regions (region A, region B, region C, and region D), where the areas of the four sub-regions are equal. Region A represents the area near the left wheel of the vehicle, region B represents the area near the right wheel of the vehicle, region C represents the area far from the left wheel of the vehicle, and region D represents the area far from the right wheel of the vehicle.

[0078] S4212 identifies the sub-regions and obtains the first road surface type and its proportion for each sub-region.

[0079] As an example, the first road surface type corresponding to a sub-region can be the percentage of multiple road surface types. For example: snow - 0%, mud - 25%, grass - 47%, sand - 12%, ordinary road surface - 0%, and non-road surface - 16%.

[0080] Specifically, based on the second road surface type and the first road surface type identified by the deep learning model and their proportions, the road surface types are obtained, including:

[0081] S4221, obtain the second road surface type corresponding to the target sub-region, and the second road surface type of the target sub-region identified by the dynamics module at the previous K time steps.

[0082] As an example, K can be 2, meaning the number of second road surface types is 3.

[0083] S4222, obtain the first road surface type and its proportion corresponding to the target sub-region, and the first road surface type and its proportion corresponding to the next sub-region of the target sub-region.

[0084] It should be noted that, referring to Figure 3, if the target sub-region is region A, then the next sub-region corresponding to the target sub-region is region C.

[0085] S4223, based on the obtained multiple second road surface types and multiple first road surface types, obtain the road surface type.

[0086] More specifically, based on the obtained multiple second pavement types and multiple first pavement types, a pavement type is obtained, including:

[0087] S42231, weights the proportions of multiple first road surface types and weights the proportions of multiple second road surface types.

[0088] As an example, when K is 2, what is obtained is the road surface type of the target sub-region identified by the dynamics module at the current moment and the road surface types identified at the two moments prior to the current moment. Let the weight coefficients corresponding to the current moment, the previous moment, and the moment before that be a, b, and c. For example, a can be 0.5, b can be 0.3, and c can be 0.2. A weighted representation of multiple second road surface types is then applied, assigning weight coefficients a, b, and c to the three obtained second road surface types respectively.

[0089] As another example, taking the target sub-region as region A, we can denote the weight coefficient corresponding to region A as weight coefficient 1 and the weight coefficient corresponding to region C as weight coefficient 2. For example, weight coefficient 1 is 0.6 and weight coefficient 2 is 0.4. By weighting the proportions of multiple first road surface types, the two resulting first road surface types are assigned weight coefficient 1 and weight coefficient 2 respectively.

[0090] It should be noted that the sum of the multiple weight coefficients corresponding to multiple second road surface types is 1, and the sum of the multiple weight coefficients corresponding to multiple first road surface types is also 1.

[0091] S42232, Based on the results of the two weighted calculations, the road surface type is obtained.

[0092] As an example, the weighting coefficients for the same road surface type in the two weighted results are added together, and the road surface type with the largest weighting coefficient is selected as the final output. For example, the weighted result for multiple second road surface types is snow 0.5, snow 0.3, and ordinary road surface 0.2; the weighted result for the proportion of multiple first road surface types is snow 0.6 and ordinary road surface 0.4. Therefore, we get snow 1.4, ordinary road surface 0.6, mud 0, sand 0, and grass 0. Thus, the output road surface type at this moment is snow.

[0093] As an example, the final road surface type can be obtained by calculating the weighted results of the two calculations through the vehicle's arbitration module.

[0094] In this embodiment, the road surface type is obtained by analyzing multiple second road surface types and multiple first road surface types, which can improve the accuracy of road surface identification.

[0095] In some embodiments, the semantic segmentation model is obtained in the following ways:

[0096] S221, acquire first image data, the first image data including road surface area and non-road surface area.

[0097] It should be noted that the first image data can be collected through vehicle-mounted cameras and / or handheld cameras. The first image data should cover as many road types, lighting conditions, and weather conditions as possible to ensure the generalization ability of the semantic segmentation model.

[0098] S222, Label the road surface and non-road surface in the first image data to obtain the first training set.

[0099] As an example, the first image data can be labeled manually, so that the first training set includes multiple labeled first image data.

[0100] S223, The semantic segmentation model is trained using the first training set to obtain a trained semantic segmentation model.

[0101] As an example, the training process of a semantic segmentation model may include steps such as forward propagation, loss calculation, backpropagation, and weight update. Through multiple iterations, the semantic segmentation model gradually learns the ability to extract road surface features from the input image data and continuously optimizes its parameters to reduce prediction errors.

[0102] In this embodiment, training the semantic segmentation model with first image data acquired by an in-vehicle camera and / or a handheld camera can improve the generalization ability of the semantic segmentation model.

[0103] In some embodiments, after acquiring the first image data, the training process of the semantic segmentation model further includes:

[0104] The first image data with duplicate road surface information is deleted, and / or the first image data is style-transferred to increase the number of first image data.

[0105] As an example, since the first image data can be acquired under different weather and lighting conditions, style transfer can be performed using the Paint Transformer method, allowing the first image data to be rendered under various weather and lighting conditions.

[0106] As another example, for a smooth, ordinary road surface on a highway, since the amount of data collected is large and the characteristics are similar, it is sufficient to retain only a portion of it.

[0107] In this embodiment, removing duplicate first image data of road surface information reduces data redundancy and improves model training efficiency. Since road surfaces and non-road surfaces have different characteristics, and weather and lighting conditions do not affect road surface recognition, collecting first image data under different weather and lighting conditions helps increase sample richness, resulting in better generalization performance of the trained semantic segmentation model. Simultaneously, style transfer reduces data acquisition costs.

[0108] In some embodiments, during the training of the semantic segmentation model using the first training set, data augmentation is performed on the samples in the first training set using at least one of the following methods:

[0109] Rotation, translation, shearing, flipping, noise injection, color transformation, dithering, blurring, and category blending.

[0110] It should be noted that the samples in the first training set are the first image data with labels.

[0111] In this embodiment, data augmentation is performed on the samples in the first training set using various data augmentation methods, which can improve the overall performance of the trained semantic segmentation model.

[0112] In some embodiments, the deep learning model is obtained in the following ways:

[0113] S31, acquire the second image data, which comes from the recognition device itself.

[0114] It should be noted that the recognition device itself can be a vehicle-mounted camera, and the second image data should cover as many road types, lighting conditions, and weather conditions as possible to ensure the generalization ability of the deep learning model.

[0115] S32, acquire the third image data, which comes from the network.

[0116] It should be noted that the third image data can be image data that already exists on the public network and includes road surface data.

[0117] As an example, the second and third image data can be standardized in format and size by resizing.

[0118] Specifically, the second and third image data include different types of road surfaces. The different types of road surfaces in the second and third image data are labeled to obtain the second training set.

[0119] S33, the road surface in the second and third image data is labeled to obtain the second training set.

[0120] As an example, the second and third image data can be labeled manually.

[0121] In this process, manual annotation involves setting the pixel values ​​of non-road areas in the second and third image data to 0. Mud, sand, snow, grassland, and ordinary roads are respectively labeled in red, yellow, blue, green, and gray.

[0122] S34, the deep learning model is trained using the second training set to obtain a trained deep learning model.

[0123] As an example, the training process of a deep learning model may include steps such as forward propagation, loss calculation, backpropagation, and weight update. Through multiple iterations, the deep learning model gradually learns the ability to identify road surface types from input image data and continuously optimizes its parameters to reduce prediction errors.

[0124] In this embodiment, training the deep learning model with second and third image data can improve the accuracy of the deep learning model in identifying road surface types.

[0125] In some embodiments, training the deep learning model using a second training set includes:

[0126] S341, the ACFNet model, Deeplabv3 model, and SegNet model were trained using the second training set.

[0127] As an example, for the ACFNet model, a second training set is used for training. The predicted output is calculated through forward propagation, then compared with the true label, and the loss function is calculated. Based on the value of the loss function, the weight parameters of the ACFNet model are updated using the backpropagation algorithm to optimize its performance. For the Deeplabv3 model, the training process is similar to that of the ACFNet model, using forward and backpropagation to update the weight parameters and improve its ability to segment road surfaces. For the SegNet model, a second training set is also used for training. By iteratively optimizing the weight parameters, the SegNet model can better identify road surface types.

[0128] S342 controls the weight allocation unit to weight the outputs of the ACFNet model, Deeplabv3 model, and SegNet model, and adjusts the weight values ​​of the ACFNet model, Deeplabv3 model, and SegNet model using the particle swarm optimization algorithm.

[0129] It should be noted that the sum of the weights of the ACFNet, Deeplabv3, and SegNet models is 1. These weights determine the degree to which each model contributes to the final deep learning-based type recognition result.

[0130] It is worth mentioning that the weight allocation unit finds an optimal combination of weight values, so that the outputs of the ACFNet model, Deeplabv3 model and SegNet model can be fused together most effectively.

[0131] In this embodiment, by weighting the outputs of the ACFNet model, the Deeplabv3 model, and the SegNet model, the recognition accuracy and stability of the deep learning model can be improved.

[0132] In some embodiments, before annotating the road surface in the second and third image data, the method further includes preprocessing the second and third image data.

[0133] The preprocessing involves stitching together the second and third image data to increase the number of images.

[0134] As an example, data concatenation can be done vertically, horizontally, or using the Cutmix method. These concatenation methods allow the features of the samples to have different aspect ratios when input into a deep learning model.

[0135] In addition, this application also provides a road surface recognition device, as shown in Figure 4, the road surface recognition device includes:

[0136] The data acquisition module 201 is used to acquire the image data to be recognized.

[0137] It should be noted that the image data to be recognized in the current scene can be acquired through an onboard camera. The onboard camera acts as an image sensor, used to acquire real-time images of the road surface in front of the vehicle. These real-time images can be three-channel RGB images.

[0138] As an example, the vehicle-mounted camera could be a monocular camera.

[0139] The first data processing module 202 is used to segment the road surface information in the image data to be recognized to obtain segmented image data.

[0140] The second data processing module 203 is used to identify the segmented image data and obtain the first road surface type.

[0141] The road surface recognition device of this invention can identify the road surface type of the current scene in real time, providing an important basis for autonomous driving decision-making. At the same time, by identifying the road surface type in two stages, the accuracy of road surface recognition is improved.

[0142] In addition, this application also provides a computer program product, including a computer program that implements the above-described road surface recognition method when executed by a processor.

[0143] The computer program product of this invention can identify the road surface type in the current scene in real time by implementing the road surface recognition method, which provides an important decision basis for autonomous driving. At the same time, the accuracy of road surface recognition is improved by identifying the road surface type in two stages.

[0144] In addition, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described road surface recognition method.

[0145] The computer-readable storage medium of this invention enables the real-time identification of road surface types in the current scene by implementing a road surface recognition method, providing an important basis for decision-making in autonomous driving. At the same time, the accuracy of road surface recognition is improved by identifying road surface types in two stages.

[0146] In addition, this application also provides a controller, as shown in FIG5. The controller 100 includes a memory 101, a processor 102, and a computer program stored in the memory 101. The memory 101 and the processor 102 are connected, for example, via a bus 103. Optionally, the controller 100 may further include a transceiver 104. When the computer program is executed by the processor 102, it implements the aforementioned road surface recognition method. It should be noted that in practical applications, the transceiver 104 is not limited to one type, and the structure of the controller 100 does not constitute a limitation on the embodiments of the present invention.

[0147] Processor 102 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 102 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0148] Bus 103 may include a pathway for transmitting information between the aforementioned components. Bus 103 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 103 may be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 5, but this does not mean that there is only one bus or one type of bus.

[0149] The memory 101 stores a computer program corresponding to the road surface recognition method of the above embodiments of the present invention. This computer program is controlled and executed by the processor 102. The processor 102 executes the computer program stored in the memory 101 to implement the content shown in the aforementioned road surface recognition method embodiments.

[0150] The controller 100 includes, but is not limited to, mobile terminals such as PDAs (Personal Digital Assistants) and PADs (Tablet PCs). The controller 100 shown in Figure 5 is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention.

[0151] The controller 100 of this invention segments the image data to be identified using a semantic segmentation model, and then identifies the segmented image data using a deep learning model. This enables real-time identification of the road surface type in the current scene, providing an important decision-making basis for autonomous driving. At the same time, the two-stage road surface type identification improves the accuracy of road surface identification.

[0152] In addition, this application also provides a vehicle, as shown in FIG6, the vehicle 10 including the controller 100 described above.

[0153] The vehicle 10 of this embodiment of the invention, through the configured controller 100, can identify the road surface type of the current scene in real time, providing an important basis for autonomous driving decision-making. At the same time, by identifying the road surface type in two stages, the accuracy of road surface identification is improved.

[0154] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0157] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0158] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0159] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0160] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0161] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0162] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0163] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A road surface recognition method, characterized in that, The method includes: acquiring image data to be identified; segmenting road surface information in the image data to be identified using a semantic segmentation model to obtain segmented image data, wherein the segmentation includes: inputting the image data to be identified into a semantic segmentation model to segment out a region of interest including a road surface area and a region of non-interest including a non-road surface area; setting the pixel values ​​of pixels in the region of non-interest to preset values ​​to obtain the segmented image data including the region of interest and the region of non-interest; and using a deep learning model to identify the segmented image data to obtain a first road surface type; wherein the deep learning model includes an ACFNet model, a Deeplabv3 model, and a SegNet model, and the deep learning model further includes a weight allocation unit, the weight allocation unit being connected to the ACFNet model, the Deeplabv3 model, and the SegNet model respectively. The model is connected to the SegNet model; an improved ResNet network is used as the backbone network of the ACFNet model, an improved RepVGG network is used as the backbone network of the Deeplabv3 model, and an improved RegNet network is used as the backbone network of the SegNet model; the improved ResNet network includes a Coordinate attention mechanism, which is set between the third and fourth convolutional layers of the ResNet network; the improved RepVGG network includes a CBAM attention mechanism, which is set in the output stage of the RepVGG network; the improved RegNet network includes an FcaNet attention mechanism, which is set in various stages of the RegNet network.

2. The road surface recognition method according to claim 1, characterized in that, The semantic segmentation model includes the LeakyReLU activation function.

3. The road surface recognition method according to claim 1, characterized in that, After recognizing the segmented image data using a deep learning model, the method further includes: using a dynamics module to obtain a second road surface type for the region corresponding to the image data to be recognized; and obtaining the road surface type based on the second road surface type and the first road surface type and its proportion identified by the deep learning model.

4. The road surface recognition method according to claim 3, characterized in that, The process of the deep learning model recognizing the segmented image data includes: dividing the segmented image data into multiple sub-regions; recognizing the sub-regions to obtain the first road surface type and its proportion corresponding to each sub-region.

5. The road surface recognition method according to claim 4, characterized in that, The step of obtaining the road surface type based on the second road surface type and the first road surface type and its proportion identified by the deep learning model includes: obtaining the second road surface type corresponding to the target sub-region, and the second road surface type of the target sub-region identified by the dynamics module at the previous K time steps; obtaining the first road surface type and its proportion corresponding to the target sub-region, and the first road surface type and its proportion of the next sub-region corresponding to the target sub-region; and obtaining the road surface type based on the obtained multiple second road surface types and multiple first road surface types.

6. The road surface recognition method according to claim 5, characterized in that, The step of obtaining the road surface type based on the multiple second road surface types and the multiple first road surface types includes: weighting the proportions of the multiple first road surface types and weighting the multiple second road surface types; and obtaining the road surface type based on the results of the two weightings.

7. The road surface recognition method according to claim 1, characterized in that, The semantic segmentation model is obtained by: acquiring first image data, which includes road surface regions and non-road surface regions; labeling the road surface and non-road surface regions in the first image data to obtain a first training set; and training the semantic segmentation model using the first training set to obtain a trained semantic segmentation model.

8. The road surface recognition method according to claim 7, characterized in that, After acquiring the first image data, the training process of the semantic segmentation model further includes: deleting the first image data with repeated road surface information, and / or performing style transfer processing on the first image data to increase the number of the first image data.

9. The road surface recognition method according to claim 7, characterized in that, During the training of the semantic segmentation model using the first training set, data augmentation is performed on the samples in the first training set using at least one of the following methods: rotation, translation transformation, shearing transformation, flipping, noise injection, color transformation, dithering, blurring, and category mixing.

10. The road surface recognition method according to claim 1, characterized in that, The method for obtaining the deep learning model includes: acquiring second image data, which comes from the recognition device itself; acquiring third image data, which comes from the network; annotating the road surface in the second image data and the third image data to obtain a second training set; and training the deep learning model using the second training set to obtain a trained deep learning model.

11. The road surface recognition method according to claim 10, characterized in that, The second image data and the third image data include different types of road surfaces. The different types of road surfaces in the second image data and the third image data are labeled to obtain a second training set.

12. The road surface recognition method according to claim 10, characterized in that, The step of training the deep learning model using the second training set includes: training the ACFNet model, the Deeplabv3 model, and the SegNet model respectively using the second training set; controlling the weight allocation unit to weight the outputs of the ACFNet model, the Deeplabv3 model, and the SegNet model, and adjusting the weight values ​​of the ACFNet model, the Deeplabv3 model, and the SegNet model using a particle swarm optimization algorithm.

13. The road surface recognition method according to claim 10, characterized in that, Before annotating the road surface in the second image data and the third image data, the method further includes: preprocessing the second image data and the third image data; wherein, the preprocessing is to perform data stitching processing on the second image data and the third image data to increase the number of the second image data and the third image data.

14. A road surface recognition device, characterized in that, The device includes: a data acquisition module for acquiring image data to be recognized; a first data processing module for segmenting road surface information in the image data to be recognized to obtain segmented image data, wherein the segmentation includes: inputting the image data to be recognized into a semantic segmentation model to segment out a region of interest including a road surface region and a region of non-interest including a non-road surface region; setting the pixel values ​​of pixels in the region of non-interest to a preset value to obtain the segmented image data including the region of interest and the region of non-interest; and a second data processing module for recognizing the segmented image data to obtain a first road surface type; wherein the second data processing module uses a deep learning model to recognize the segmented image data to obtain the first road surface type, the deep learning model including an ACFNet model, a Deeplabv3 model, and a SegNet model, and the deep learning model further includes a weight allocation unit, wherein the weight allocation... The unit is connected to the ACFNet model, the Deeplabv3 model, and the SegNet model, respectively. An improved ResNet network is used as the backbone network of the ACFNet model, an improved RepVGG network as the backbone network of the Deeplabv3 model, and an improved RegNet network as the backbone network of the SegNet model. The improved ResNet network includes a Coordinate attention mechanism, which is set between the third and fourth convolutional layers of the ResNet network. The improved RepVGG network includes a CBAM attention mechanism, which is set in the output stage of the RepVGG network. The improved RegNet network includes an FcaNet attention mechanism, which is set in various stages of the RegNet network.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the road surface recognition method according to any one of claims 1-13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the road surface recognition method according to any one of claims 1-13.

17. A controller comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is executed by the processor, it implements the road surface recognition method according to any one of claims 1-13.

18. A vehicle, characterized in that, Includes the controller as described in claim 17.

Images