Road surface type detection method, device, vehicle, electronic equipment and storage medium

Abstract

The embodiments of the present invention provide a road surface type detection method, device, vehicle, electronic device and storage medium. The method includes: acquiring a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining the road surface type corresponding to the wheel track area based on a pre-trained deep learning model. The embodiments of the present invention can reduce the amount of image data to be processed by preprocessing the real-time image to obtain a receptive field image, and determine the wheel track area from the receptive field image, so that the detection method can focus on the road surface information within the area of ​​interest without having to process the entire scene, thereby reducing computational complexity. By identifying more types of road surface types through a deep learning model, the material type of the road surface in front of the vehicle can be accurately identified in real time, and high computational efficiency and real-time performance are ensured.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a road surface type detection method, a road surface type detection device, an autonomous driving vehicle, an electronic device and a storage medium. Background Art

[0002] With the rapid development of autonomous driving technology, it is crucial to ensure that vehicles can drive smoothly under different road conditions. During the autonomous driving process, the vehicle needs to identify the type of road ahead in real time so that the vehicle's driving behavior can be adjusted according to the material type of the road. Different road types, such as cement, asphalt, snow, ice, dirt roads, grass, water, rocky roads, and deserts, have different requirements for vehicle handling and performance. For example, on snow or ice, the vehicle's speed needs to be reduced and grip needs to be improved; on asphalt roads, a higher speed and stability can be maintained. Therefore, road type detection is crucial to the performance of the autonomous driving system.

[0003] In the prior art, sensor data and computer vision technology are used to detect the road surface type, and computer vision technology, such as edge detection, color analysis, texture analysis, etc., and empirical rules are used to identify the road surface type. Sensor data detection, such as laser radar or millimeter wave radar, is used to obtain road surface information.

[0004] However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] Most existing technologies have limitations in road surface type detection and can only detect basic road surface types, such as cement or asphalt, but have difficulty distinguishing more complex road surface types, such as snow, ice, and waterlogged areas. In addition, existing technologies usually need to obtain a large amount of terrain data from multiple sensors, including image data and point cloud data. These data need to be complexly processed and fused to generate accurate terrain information, which increases the complexity of algorithms and calculations and cannot cope with rapidly changing real-time road conditions. Summary of the invention

[0006] In view of the defects in the prior art, embodiments of the present invention provide a road type detection method, a road type detection device, an automatic driving vehicle, an electronic device and a storage medium.

[0007] In a first aspect, a road surface type detection method is provided, comprising:

[0008] Get real-time images transmitted by the vehicle camera;

[0009] Preprocessing the real-time image to determine a receptive field image;

[0010] determining a wheel track region from the receptive field image;

[0011] According to a pre-trained deep learning model, the road surface type corresponding to the wheel track area is determined.

[0012] As in the above method, optionally, determining the road surface type corresponding to the wheel track area according to a pre-trained deep learning model includes:

[0013] Determining a pixel category of each pixel on the receptive field image based on a pre-trained semantic segmentation model;

[0014] According to the pixel category of each pixel point in the wheel track area, the road surface type corresponding to the wheel track area is determined.

[0015] As in the above method, optionally, the semantic segmentation model is trained by the following steps:

[0016] Obtain sample receptive field images, and mark the pixel category of each pixel point according to the road surface type of each pixel point in each sample receptive field image to obtain a receptive field image sample set;

[0017] Dividing the receptive field image sample set into a receptive field image training set and a receptive field image test set;

[0018] Build a semantic segmentation model based on a lightweight backbone network;

[0019] Input each sample receptive field image in the receptive field image training set into the semantic segmentation model, perform multi-scale semantic feature extraction on the sample receptive field image through the semantic segmentation model, and output the pixel category of each pixel point;

[0020] Based on the receptive field image test set, the semantic segmentation model is optimized.

[0021] As in the above method, optionally, determining the road surface type corresponding to the wheel track area according to the pixel category of each pixel point in the wheel track area includes:

[0022] Traversing the pixel categories of each row of pixels in the wheel track area by column;

[0023] Count the number of pixels in each pixel category by row, determine the weight of each pixel category according to the total number of pixels in each row, and obtain the normalized number of each pixel category;

[0024] After the traversal is completed, the normalized number of each pixel category in the wheel track area is sorted, and the pixel category with the largest normalized number is used as the road surface type corresponding to the wheel track area.

[0025] As in the above method, optionally, determining the road surface type corresponding to the wheel track area according to a pre-trained deep learning model includes:

[0026] Preprocessing the wheel track area into a wheel track rectangular area;

[0027] Inputting the wheel track rectangular area into a pre-trained multi-label classification network to determine the confidence of each road surface type in the wheel track rectangular area;

[0028] The road surface type corresponding to the wheel track area is determined according to the confidence level.

[0029] As in the above method, optionally, the multi-label classification network is trained by the following steps:

[0030] Obtaining sample wheel track rectangular images, marking multiple road surface types corresponding to each sample wheel track rectangular image, and obtaining a wheel track rectangular image sample set;

[0031] Dividing the wheel track rectangular image sample set into a wheel track rectangular image training set and a wheel track rectangular image test set;

[0032] Based on the deep convolutional neural network model, a multi-label classification network is constructed;

[0033] Inputting each sample wheel track rectangular image in the wheel track rectangular image training set into the multi-label classification network, performing multi-layer convolution through the multi-label classification network, and outputting the road surface type and corresponding confidence of each sample wheel track rectangular image;

[0034] Based on the wheel track rectangle image test set, the multi-label classification network is optimized.

[0035] As in the above method, optionally, after determining the confidence level of each road surface type in the wheel track rectangular area, the method further includes:

[0036] Determine the probability of each road surface type based on the confidence level of each road surface type;

[0037] Output each road surface type and the corresponding probability corresponding to the wheel track area.

[0038] As in the above method, optionally, determining the wheel track area from the receptive field image includes:

[0039] Determine the vehicle's front direction and tire status based on the vehicle's current driving trajectory and predicted trajectory;

[0040] According to the vehicle head orientation and tire status, a left wheel track area and a right wheel track area are determined from the receptive field image.

[0041] As in the above method, optionally, the method further comprises:

[0042] The road surface type corresponding to the wheel track area is sent to a vehicle control device so that the vehicle control device controls the vehicle according to the road surface type.

[0043] In a second aspect, a road surface type detection device is provided, comprising:

[0044] An acquisition module is used to acquire real-time images transmitted by the vehicle-mounted camera;

[0045] A preprocessing module, used to preprocess the real-time image to determine a receptive field image;

[0046] A wheel track extraction module, used to determine the wheel track area from the receptive field image;

[0047] The recognition module is used to determine the road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0048] In a third aspect, an autonomous driving vehicle is provided, comprising: a road surface type detection device, wherein the road surface type detection device is used to execute a road surface type detection method as described in any one of the first aspects above.

[0049] In a fourth aspect, an electronic device is provided, including:

[0050] A memory and a processor, wherein the processor and the memory communicate with each other via a bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the following method: acquiring a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0051] In a fifth aspect, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following method is implemented: obtaining a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0052] One of the above technical solutions has the following advantages:

[0053] Road type detection based on real-time images transmitted by on-board cameras reduces dependence on expensive sensors and helps reduce the development and maintenance costs of autonomous driving systems. By preprocessing real-time images and obtaining receptive field images, the amount of image data processed can be reduced, the efficiency of road type detection can be improved, and it is easy to apply to real-time road type detection. Furthermore, the wheel track area is determined from the receptive field image, so that the detection method can focus on the road information in the area of ​​interest without having to process the entire scene, reducing the computational complexity. Finally, the road type corresponding to the wheel track area is identified through a pre-trained deep learning model. By identifying more types of road types through the deep learning model, the material type of the road in front of the vehicle can be accurately identified in real time, and high computational efficiency and real-time performance are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of steps of an embodiment of a road surface type detection method of the present invention;

[0055] Figure 2 is a real-time image schematic diagram of an embodiment of a road surface type detection method of the present invention;

[0056] Figure 3 is a schematic diagram of a road surface type detection result in an embodiment of a road surface type detection method of the present invention;

[0057] Figure 4 is another schematic diagram of a road surface type detection result in an embodiment of a road surface type detection method of the present invention;

[0058] Figure 5 is a schematic diagram of another road surface type detection result in an embodiment of a road surface type detection method of the present invention;

[0059] Figure 6 is a schematic diagram of another road surface type detection result in an embodiment of a road surface type detection method of the present invention;

[0060] Figure 7 is a structural block diagram of an embodiment of a road surface type detection device of the present invention;

[0061] Figure 8 It is a structural block diagram of an electronic device embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] In the prior art, sensor data and computer vision technology are used to detect the road surface type, and computer vision technology, such as edge detection, color analysis, texture analysis, etc., and empirical rules are used to identify the road surface type. Sensor data detection, such as laser radar or millimeter wave radar, is used to obtain road surface information.

[0064] However, most existing technologies have limitations in road surface type detection and can only detect basic road surface types, such as cement or asphalt, but have difficulty distinguishing more complex road surface types, such as snow, ice, and waterlogged areas. In addition, existing technologies usually need to obtain a large amount of terrain data from multiple sensors, including image data and point cloud data. These data need to be complexly processed and fused to generate accurate terrain information, which increases the complexity of algorithms and calculations and cannot cope with rapidly changing real-time road conditions.

[0065] In order to solve this problem, an embodiment of the present invention obtains a real-time image transmitted by a vehicle-mounted camera; preprocesses the real-time image to determine a receptive field image; determines a wheel track area from the receptive field image; and determines a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0066] The embodiment of the present invention detects the road type based on the real-time image transmitted by the vehicle-mounted camera, reduces the dependence on expensive sensors, and helps to reduce the development and maintenance costs of the autonomous driving system; by preprocessing the real-time image and obtaining the receptive field image, the amount of image data to be processed can be reduced, the efficiency of road type detection can be improved, and it is easy to apply to real-time road type detection. Furthermore, the wheel track area is determined from the receptive field image, so that the detection method can focus on the road surface information in the area of ​​interest without having to process the entire scene, reducing the computational complexity. Finally, the road surface type corresponding to the wheel track area is identified through a pre-trained deep learning model. By identifying more types of road surface types through the deep learning model, the material type of the road surface in front of the vehicle can be accurately identified in real time, and high computational efficiency and real-time performance are ensured.

[0067] Reference Figure 1 , shows a flow chart of a road surface type detection method embodiment of the present invention, which may specifically include the following steps:

[0068] Step S110, obtaining a real-time image transmitted by the vehicle-mounted camera;

[0069] Specifically, during the autonomous driving process, the vehicle needs to identify the type of road ahead in real time so that the vehicle's driving behavior can be adjusted according to the material type of the road. Different road types, such as cement, asphalt, snow, ice, dirt roads, grass, water, rocky roads, and deserts, have different requirements for vehicle handling and performance. Therefore, real-time and accurate identification of the material type of the road ahead is critical to the autonomous driving system. Road conditions in the real world are very diverse, and the road surface may be a mixture of multiple materials. For example, the road surface may be a dirt road in one area and asphalt in another area. The vehicle needs to be able to adapt to this diversity to ensure smooth and safe driving. In addition, different road conditions such as water, snow, and ice also require precise identification and response.

[0070] In order to detect the road type, real-time road images in front of the vehicle can be obtained. In autonomous driving technology, obtaining road information in front of the vehicle usually relies on on-board cameras, which can capture road images within a certain distance range in front of the vehicle. Figure 2 , shows a schematic diagram of a real-time image in an embodiment of a road type detection method of the present invention. It can be seen that the real-time image includes road surface information in front of the vehicle.

[0071] Step S120, preprocessing the real-time image to determine a receptive field image;

[0072] Specifically, from Figure 2 It can be seen that the real-time image transmitted by the vehicle-mounted camera includes not only the real-time image of the road surface but also the image of the sky and the image of the surrounding buildings. In order to improve the efficiency of road surface type detection, in the embodiment of the present invention, the real-time image is processed, for example, the original RAW image is converted into a red, green and blue RGB image and other conventional image processing, and then the receptive field image is cropped, such as Figure 2 The receptive field image can be cropped from the processed real-time image by using the preset receptive field rectangular area size, and the receptive field rectangular area size can be set according to the actual situation.

[0073] For example, the vehicle control device needs to predict the road conditions N meters ahead of the vehicle in order to adjust the vehicle status. The size of the receptive field rectangular area in the real-time image can be determined based on N and the imaging law of the on-board camera, and then the coordinates of the receptive field rectangular area can be determined based on the position of the vehicle's front end.

[0074] By preprocessing the real-time image transmitted by the vehicle-mounted camera to obtain the receptive field image, the size of the processed image can be reduced, thereby improving the efficiency of road type detection and facilitating its application in real-time detection.

[0075] Step S130, determining a wheel track area from the receptive field image;

[0076] Specifically, in order to reduce computational complexity, the receptive field image can be further segmented to determine the wheel track area, where the wheel track area refers to the road surface area where the wheels are about to travel. In practical applications, only the road surface type of the road surface area where the wheels are about to travel can be determined, and the vehicle can be further controlled according to the road surface type. Therefore, the wheel track area is determined from the receptive field image. This area selection enables the detection method to focus on the road surface information in the area of ​​interest without having to process the entire scene, further reducing computational complexity.

[0077] In a specific implementation, determining the wheel track area from the receptive field image includes:

[0078] Determine the vehicle's front direction and tire status based on the vehicle's current driving trajectory and predicted trajectory;

[0079] According to the vehicle head orientation and tire status, a left wheel track area and a right wheel track area are determined from the receptive field image.

[0080] Specifically, a vehicle will form a left wheel track area and a right wheel track area during operation, and the two represent the road surface area where the left and right wheels are about to travel. In order to determine the segmentation of the left wheel track area and the right wheel track area from the receptive field image, the current direction of the vehicle head and the tire status can be determined according to the current driving trajectory and the predicted trajectory of the vehicle, for example, the direction and angle of the vehicle head, the tire angle, whether the tire is ready to turn left or right, etc., so as to predict the road surface area where the left and right wheels are about to travel, and then according to the imaging rules of the on-board camera, the projections of the road surface area where the left and right wheels are about to travel are respectively mapped from the receptive field image, and the projections are used as the left wheel track area or the right wheel track area.

[0081] like Figure 2 The trapezoidal areas on the left and right sides shown are the left wheel track area and the right wheel track area respectively. According to the rule that objects are larger when closer and smaller when farther away, the left wheel track area and the right wheel track area are a trapezoidal area, and the long sides of the trapezoidal areas are close to the front of the vehicle, and the short sides are far away from the front of the vehicle.

[0082] The embodiment of the present invention obtains image data of the road ahead through a vehicle-mounted camera, and limits the processed image to a certain distance range in front of the vehicle. During the image acquisition process, corresponding projection and analysis are performed based on the information of the left and right wheel tracks to determine the left wheel track area and the right wheel track area, which helps to more accurately determine the road material type of the entire wheel track.

[0083] Step S140: Determine the road surface type corresponding to the wheel track area according to the pre-trained deep learning model.

[0084] Specifically, a deep learning model may be pre-trained, the input of the deep learning model may be a wheel track area, and the output may be a road surface type corresponding to the wheel track area, wherein the road surface type includes but is not limited to: cement, asphalt, snow, ice, dirt road, grass, waterlogged, rocky road, desert, etc. It should be noted that since the road surface type is detected in real time, the deep learning model needs to be a deep learning model that can be applied in real time.

[0085] The training process of a deep learning model can be:

[0086] Obtain sample wheel track areas, and mark the road surface type corresponding to each sample wheel track area, such as cement, asphalt, snow, ice, dirt road, grass, waterlogged, rocky road, and desert. For example, different parameters can be used to represent the above different road surface types to obtain a wheel track area sample set;

[0087] The wheel track area sample set is divided into a wheel track area training set and a wheel track area test set;

[0088] Build deep learning models;

[0089] Each sample wheel track area in the wheel track area training set is input into the deep learning model, the deep learning model is trained, and the training is stopped after it is determined that the cross entropy between the road surface type of the sample wheel track area output by the deep learning model and the labeled road surface type meets the set conditions or the training completes a certain number of iterations, thereby obtaining various parameters of the deep learning model.

[0090] Afterwards, each sample wheel track area in the wheel track area test set is input into the deep learning model, and the deep learning model is optimized according to the output road surface type.

[0091] After optimization, the required deep learning model is obtained. Since the deep learning model can process large-scale data and the performance of the deep learning model will become better as the data size increases, more types of road surfaces can be identified through the deep learning model with higher recognition accuracy.

[0092] After identifying the road type, the onboard system can quickly process the image of the road ahead and make quick decisions to adapt to changing road conditions.

[0093] A road surface type detection method provided by an embodiment of the present invention performs road surface type detection based on real-time images transmitted by a vehicle-mounted camera, reduces reliance on expensive sensors, and helps reduce the development and maintenance costs of autonomous driving systems; by preprocessing real-time images and obtaining receptive field images, the amount of image data processed can be reduced, the efficiency of road surface type detection can be improved, and it is easy to apply to real-time road surface type detection. Furthermore, the wheel track area is determined from the receptive field image, so that the detection method can focus on the road surface information within the area of ​​interest without having to process the entire scene, reducing computational complexity. Finally, the road surface type corresponding to the wheel track area is identified through a pre-trained deep learning model. By identifying more types of road surface types through the deep learning model, the material type of the road surface in front of the vehicle can be accurately identified in real time, and high computational efficiency and real-time performance are ensured.

[0094] On the basis of the above embodiment, further, determining the road surface type corresponding to the wheel track area according to the pre-trained deep learning model includes:

[0095] Determining a pixel category of each pixel on the receptive field image based on a pre-trained semantic segmentation model;

[0096] According to the pixel category of each pixel point in the wheel track area, the road surface type corresponding to the wheel track area is determined.

[0097] Specifically, semantic segmentation can be used to identify the road surface type. This process includes the following steps:

[0098] Step A1, obtaining a real-time image transmitted by a vehicle-mounted camera;

[0099] Step A2: preprocess the real-time image to determine the receptive field image;

[0100] Step A3, determining the left wheel track area and the right wheel track area from the receptive field image;

[0101] Step A4: Use the pre-trained semantic segmentation model to determine the pixel category of each pixel on the receptive field image. It should be noted that the pixel category corresponds to the road surface type, such as cement, asphalt, snow, ice, dirt road, grass, water, rocky road, desert, etc.

[0102] Step A5: Voting is performed according to the pixel category of each pixel point in the left wheel track area / right wheel track area to determine the road surface type corresponding to the left wheel track area / right wheel track area.

[0103] The road type detection method provided by an embodiment of the present invention can accurately identify the material type of the road ahead in real time based on a pre-trained semantic segmentation model, including cement, asphalt, snow, ice, dirt road, grass, water, rocky road, desert, etc. It provides key environmental perception information for autonomous driving vehicles and helps them make appropriate driving decisions to ensure safe and smooth driving.

[0104] On the basis of the above embodiments, further, the semantic segmentation model is trained by the following steps:

[0105] Obtain sample receptive field images, and mark the pixel category of each pixel point according to the road surface type of each pixel point in each sample receptive field image to obtain a receptive field image sample set;

[0106] Dividing the receptive field image sample set into a receptive field image training set and a receptive field image test set;

[0107] Build a semantic segmentation model based on a lightweight backbone network;

[0108] Input each sample receptive field image in the receptive field image training set into the semantic segmentation model, perform multi-scale semantic feature extraction on the sample receptive field image through the semantic segmentation model, and output the pixel category of each pixel point;

[0109] Based on the receptive field image test set, the semantic segmentation model is optimized.

[0110] Specifically, a large number of receptive field images are first obtained as sample receptive field images. According to the road surface type of each pixel in each sample receptive field image, the pixel category of each pixel is labeled to obtain a sample set of receptive field images. For example, in practical applications, an image with three channels of red, green and blue (RGB) can be expressed as w*h*3, where w and h represent the width and height of the image, and 3 represents the values ​​of the three dimensions RGB. The pixel category of each pixel in the image is labeled, which can be expressed as w*h*1, where the position 1 is used to label the pixel category of each pixel.

[0111] Afterwards, the receptive field image sample set is divided into a receptive field image training set and a receptive field image test set, where the intersection of the receptive field image training set and the receptive field image test set is empty.

[0112] Through the above process, a supervised sample set can be obtained.

[0113] Afterwards, a semantic segmentation model is built based on the lightweight backbone network. It should be noted that since the road type detection is detected in real time during autonomous driving, the semantic segmentation model needs to be applicable to real-time scenarios. For example, the semantic segmentation model can be a dual-channel network structure (Bilateral Segmentation Network, BiSeNet V2). The BiSeNet V2 network model has a lightweight network and better real-time performance. It does not need to consider information other than the road surface and can be deployed on the vehicle side. A lightweight backbone network is used to extract high-level semantic features of the input image and detailed information of the image, such as texture information.

[0114] Afterwards, each sample receptive field image in the receptive field image training set is input into the semantic segmentation model, and the semantic segmentation model is trained so that it can output the correct pixel category. Specifically, the semantic segmentation model extracts high-level semantic features and detail information of the sample receptive field image, upsamples the high-level semantic features at multiple scales (feature maps of different resolutions), and effectively fuses the branches of detail information of the sample receptive field image, thereby learning the pixel category of each pixel in the sample receptive field image. This helps the semantic segmentation model better understand the global and local structure of the sample receptive field image, improves the accuracy of the semantic segmentation model, and takes into account the real-time performance of the semantic segmentation model.

[0115] The pixel category of each pixel in the receptive field image of each sample in the receptive field image training set output by the semantic segmentation model is compared with the labeled category, and the cross entropy loss function is determined. The semantic segmentation model is optimized by the cross entropy loss function. As the model is iteratively trained, the weight parameters are updated so that the overall loss value gradually decreases.

[0116] Finally, the semantic segmentation model is tested using the receptive field image test set, and optimized based on the test results. The optimized semantic segmentation model can be deployed on the autonomous driving vehicle side for real-time identification of road surface types.

[0117] A road type detection method provided by an embodiment of the present invention uses a lightweight backbone network to build a semantic segmentation model, which helps the semantic segmentation model to better understand the global and local structures of the sample receptive field image, improves the accuracy of the semantic segmentation model and takes into account the real-time performance of the semantic segmentation model. A lightweight backbone network and multi-scale feature fusion are used to ensure high computational efficiency and real-time performance, so that the vehicle-mounted system can quickly process the road image ahead and make decisions quickly to adapt to changing road conditions.

[0118] On the basis of the above embodiments, further, determining the road surface type corresponding to the wheel track area according to the pixel category of each pixel point in the wheel track area includes:

[0119] Traversing the pixel categories of each row of pixels in the wheel track area by column;

[0120] Count the number of pixels in each pixel category by row, determine the weight of each pixel category according to the total number of pixels in each row, and obtain the normalized number of each pixel category;

[0121] After the traversal is completed, the normalized number of each pixel category in the wheel track area is sorted, and the pixel category with the largest normalized number is used as the road surface type corresponding to the wheel track area.

[0122] Specifically, for the left wheel track area or the right wheel track area, the pixel categories of each row of pixels in the wheel track area are traversed column by column, the number of pixels detected as a certain pixel category is counted row by row in the wheel track area, and then divided by the total number of pixels in the row as the weight of the pixel category, thereby eliminating the interference of more pixels near the vehicle camera and fewer pixels far away.

[0123] After the traversal is completed, the normalized number of each pixel category is counted and sorted, and the pixel category with the largest normalized number of pixels is selected as the road surface type corresponding to the wheel track.

[0124] Figure 3 FIG. 1 is a schematic diagram of a road surface type detection result in an embodiment of a road surface type detection method of the present invention. Figure 3 As shown, through the semantic segmentation model, it is determined that the road surface type of the left wheel track area is asphalt (liqing), the road surface type of the right wheel track area is asphalt (liqing), and the road surface types of the left and right wheel track areas are the same. Figure 4 FIG. 1 is another schematic diagram of a road surface type detection result in an embodiment of a road surface type detection method of the present invention. Figure 4 As described above, through the semantic segmentation model, it is determined that the road surface type of the left wheel track area is rock, and the road surface type of the right wheel track area is grass, and the road surface types of the left and right wheel track areas are different.

[0125] A road surface type detection method provided by an embodiment of the present invention traverses pixel categories by column for the left wheel track area or the right wheel track area, and determines pixel category weights, thereby eliminating interference from the vehicle-mounted camera's camera viewing angle having more pixels near and fewer pixels far away. The road surface type is determined by voting separately in the left and right wheel track areas, which can not only identify a single road surface material, but also cope with mixed road conditions, thereby enhancing the adaptability of the automatic driving system to complex road conditions and ensuring that the vehicle can drive normally in various road environments.

[0126] On the basis of the above embodiments, further, determining the road surface type corresponding to the wheel track area according to the pre-trained deep learning model includes:

[0127] Preprocessing the wheel track area into a wheel track rectangular area;

[0128] Inputting the wheel track rectangular area into a pre-trained multi-label classification network to determine the confidence of each road surface type in the wheel track rectangular area;

[0129] The road surface type corresponding to the wheel track area is determined according to the confidence level.

[0130] Specifically, in addition to using the semantic segmentation model to identify the road surface type, you can also use the multi-label classification method to realize the identification of road surface materials. The characteristic of multi-label itself is that it inputs one image and outputs multiple categories. The multi-label classification method includes the following steps:

[0131] Step B1, obtaining a real-time image transmitted by a vehicle-mounted camera;

[0132] Step B2, preprocessing the real-time image to determine the receptive field image;

[0133] Step B3, determining the left wheel track area and the right wheel track area from the receptive field image;

[0134] Step B4, adjusting the left wheel track area and the right wheel track area to wheel track rectangular areas of the same size;

[0135] Specifically, since the convolutional neural network input image is a rectangle, and the left wheel track area and the right wheel track area are special shapes, such as a trapezoid when the vehicle is moving straight and an arc when the vehicle is turning, the left wheel track area and the right wheel track area need to be restored to rectangular areas. The left wheel track area and the right wheel track area can be transformed into wheel track rectangular areas of preset pixels, such as a wheel track rectangular area of ​​256*256 pixels, through bilinear interpolation.

[0136] Step B5: The adjusted wheel track rectangular area is fed into a pre-trained multi-label classification network, which will output the confidence of each road surface type. Finally, the road surface category with the highest confidence is selected as the result of the identified road surface type. This method has the advantages of small model and fast speed, while reducing the difficulty of data annotation. In addition, it can also generate a large amount of multi-label data through data synthesis, thereby improving the generalization of the model.

[0137] A road type detection method provided by an embodiment of the present invention uses a multi-label classification model to adjust the left and right wheel track areas to rectangles of the same size, thereby reducing the difficulty of data annotation. At the same time, it also allows the use of data synthesis to generate a large amount of diverse training data to improve the generalization of the model. By simplifying the data annotation process and reducing the reliance on expensive sensors, it helps to reduce the development and maintenance costs of the autonomous driving system.

[0138] On the basis of the above embodiments, further, the multi-label classification network is trained by the following steps:

[0139] Obtaining sample wheel track rectangular images, marking multiple road surface types corresponding to each sample wheel track rectangular image, and obtaining a wheel track rectangular image sample set;

[0140] Dividing the wheel track rectangular image sample set into a wheel track rectangular image training set and a wheel track rectangular image test set;

[0141] Based on the deep convolutional neural network model, a multi-label classification network is constructed;

[0142] Inputting each sample wheel track rectangular image in the wheel track rectangular image training set into the multi-label classification network, performing multi-layer convolution through the multi-label classification network, and outputting the road surface type and corresponding confidence of each sample wheel track rectangular image;

[0143] Based on the wheel track rectangle image test set, the multi-label classification network is optimized.

[0144] Specifically, firstly, a large number of wheel track rectangular images are obtained as sample wheel track rectangular images, and various road surface types of each sample wheel track rectangular image are annotated to obtain a wheel track rectangular image sample set. For example, in practical applications, a red, green, blue (RGB) three-channel image can be represented as w*h*3, where w and h represent the width and height of the image, and 3 represents the value of the three-dimensional RGB. To annotate various road surface types of the sample wheel track rectangular image, a 01 vector can be used. For example, if there are 10 road surface types, the annotation result of the sample wheel track rectangular image can be represented as 1*1*10, where 1*1 represents a wheel track rectangular image, and 10 represents 10 road surface types. 1 is assigned to the position where the road surface type exists, and 0 is assigned to the position where the road surface type does not exist.

[0145] Afterwards, the wheel track rectangular image sample set is divided into a wheel track rectangular image training set and a wheel track rectangular image test set, wherein the intersection of the wheel track rectangular image training set and the wheel track rectangular image test set is empty.

[0146] Through the above process, a supervised sample set can be obtained.

[0147] Afterwards, a multi-label classification network is constructed based on the deep convolutional neural network model. Therefore, the multi-label classification network needs to be applied to real-time scenarios. For example, the multi-label classification network can be a residual neural network (ResNet), such as ResNet18. The basic architecture of the ResNet18 network is ResNet, and the network depth is 18 layers. The ResNet18 network model is lightweight and has better real-time performance. It does not need to consider information other than the road surface and can be deployed on the vehicle side. Using the multi-label classification network, multi-layer convolution is performed on the sample wheel track rectangular image to extract the sample wheel track rectangular image information.

[0148] Afterwards, each sample wheel track rectangular image in the wheel track rectangular image training set is input into the multi-label classification network, and the multi-label classification network is trained so that it can output the confidence of each road surface type. Specifically, the multi-label classification network uses multi-layer convolution to extract the image features of the sample wheel track rectangular image, and upsamples the image features at multiple scales to learn the confidence of each road surface type in the sample wheel track rectangular image.

[0149] The road surface type and confidence of each sample wheel track rectangular image in the wheel track rectangular image training set output by the multi-label classification network are used to determine the binary cross entropy loss function. The multi-label classification network is optimized by the binary cross entropy loss function. As the multi-label classification network model iteratively updates the weight parameters, the value of the loss function gradually decreases.

[0150] Finally, the multi-label classification network is tested using the receptive field image wheel track rectangle image test set, and the multi-label classification network is optimized based on the test results. The optimized multi-label classification network can be deployed on the autonomous driving vehicle side for real-time identification of road surface types.

[0151] The road type detection method provided by an embodiment of the present invention uses a deep convolutional neural network to construct a multi-label classification network, taking into account high computational efficiency and real-time performance, and can cope with rapidly changing road conditions. Through real-time road type recognition, the vehicle can better understand the surrounding environment, which improves the quality of environmental perception and helps to improve the performance and safety of the autonomous driving system.

[0152] On the basis of the above embodiments, further, after determining the confidence level of each road surface type in the wheel track rectangular area, the method further includes:

[0153] Determine the probability of each road surface type based on the confidence level of each road surface type;

[0154] Output each road surface type and the corresponding probability corresponding to the wheel track area.

[0155] Specifically, the road conditions in the real world are very diverse, and the road surface may be a mixture of multiple materials. For example, the road surface may be a dirt road in one area and asphalt in another area. The vehicle needs to be able to adapt to this diversity to ensure smooth and safe driving. Therefore, there may be multiple road surface types in the wheel track area. Therefore, after the multi-label classification network outputs the confidence of each road surface type, the probability of each road surface type can also be determined, converted into probability based on the confidence, and each road surface type and corresponding probability corresponding to the left wheel track area and each road surface type and corresponding probability corresponding to the right wheel track area are output.

[0156] Reference Figure 5 , shows another schematic diagram of a road surface type detection result in an embodiment of a road surface type detection method of the present invention, the detection result of the left wheel track area (left_traj) is: asphalt (liqing) 0.9939739, snow (snow) 0.12683805, soil (soil) 0.31089333; the detection result of the right wheel track area (right_traj) is: asphalt (liqing) 0.8894275, snow (snow) 0.122526854, soil (soil) 0.6719509. It can be seen that the detection result has a high accuracy rate.

[0157] Reference Figure 6 , shows another schematic diagram of road type detection results in an embodiment of a road type detection method of the present invention, the detection results of the left wheel track area (left_traj) are: asphalt (liqing) 0.4481995, snow (snow) 0.99985695, soil (soil) 0.23830785; the detection results of the right wheel track area (right_traj) are: asphalt (liqing) 0.45402208, snow (snow) 0.99869967, soil (soil) 0.16864307. It can be seen that the detection result has a high accuracy rate.

[0158] The road type detection method provided by an embodiment of the present invention can not only identify a single road material, but also cope with mixed road conditions, such as when there are multiple materials on the road surface. It can effectively identify multiple material types of the road ahead, provide key environmental perception information for the automatic driving system, enhance the adaptability of the automatic driving system to complex road conditions, and ensure that the vehicle can drive normally in various road environments.

[0159] In an embodiment of the present invention, the road surface type detection can be implemented by a road surface type detection method based on a semantic segmentation model, which mainly includes the following steps:

[0160] Step A1, obtaining a real-time image transmitted by a vehicle-mounted camera;

[0161] Step A2: preprocess the real-time image to determine the receptive field image;

[0162] Step A3, determining the left wheel track area and the right wheel track area from the receptive field image;

[0163] Step A4: Use the pre-trained semantic segmentation model to determine the pixel category of each pixel on the receptive field image. It should be noted that the pixel category corresponds to the road surface type, such as cement, asphalt, snow, ice, dirt road, grass, water, rocky road, desert, etc.

[0164] Step A5: Voting is performed according to the pixel category of each pixel point in the left wheel track area / right wheel track area to determine the road surface type corresponding to the left wheel track area / right wheel track area.

[0165] In addition, the road type detection can also be implemented based on the road type detection method of the multi-label classification network, which mainly includes the following steps:

[0166] Step B1, obtaining a real-time image transmitted by a vehicle-mounted camera;

[0167] Step B2, preprocessing the real-time image to determine the receptive field image;

[0168] Step B3, determining the left wheel track area and the right wheel track area from the receptive field image;

[0169] Step B4, adjusting the left wheel track area and the right wheel track area to wheel track rectangular areas of the same size;

[0170] Step B5: The adjusted wheel track rectangular area is fed into a pre-trained multi-label classification network, which outputs the confidence of each road surface type. Finally, the road surface category with the highest confidence is selected as the result of the identified road surface type.

[0171] When making a specific choice, you can choose the road surface type detection method based on the semantic segmentation model or the road surface type detection method based on the multi-label classification network according to computing power and usage requirements.

[0172] In addition, the semantic segmentation model can be combined with the multi-label classification network to jointly determine the road surface types in the left wheel track area and the right wheel track area, further improving the robustness of road surface type recognition.

[0173] For example, a first weight corresponding to the semantic segmentation model and a second weight corresponding to the multi-label classification network can be set, and various recognized results can be multiplied by the corresponding weights to determine the probability of the road type. Finally, the road type with the highest probability value is the final recognition result.

[0174] Alternatively, the first result identified by the semantic segmentation model can be combined with the confidence of various road surface types output by the multi-label classification network to jointly determine the road surface type.

[0175] Alternatively, other combinations may be provided, which will not be described in detail here.

[0176] On the basis of the above embodiments, the method further includes:

[0177] The road surface type corresponding to the wheel track area is sent to a vehicle control device so that the vehicle control device controls the vehicle according to the road surface type.

[0178] Specifically, the road surface type corresponding to the identified wheel track area is sent to the vehicle control device, which controls the vehicle according to the road surface type to adjust the vehicle's suspension, steering, braking and other parameters to ensure the vehicle's driving stability and safety on different road surfaces. For example, on snow or ice, the vehicle's speed needs to be reduced and its grip needs to be improved; on asphalt roads, a higher speed and stability can be maintained.

[0179] A road type detection method provided by an embodiment of the present invention realizes real-time perception and output of the road type ahead of the vehicle, so that the autonomous driving system can adapt to different road materials, provide a smooth driving experience, and promote the further development of autonomous driving technology, making it more suitable for diverse roads and environments.

[0180] Reference Figure 7 , shows a structural block diagram of an embodiment of a road type detection device of the present invention, which may specifically include the following modules: an acquisition module 710, a preprocessing module 720, a wheel track extraction module 730 and an identification module 740, wherein:

[0181] An acquisition module 710 is used to acquire a real-time image transmitted by a vehicle-mounted camera;

[0182] A preprocessing module 720 is used to preprocess the real-time image to determine a receptive field image;

[0183] A wheel track extraction module 730, used to determine the wheel track area from the receptive field image;

[0184] The identification module 740 is used to determine the road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0185] As in the above device, optionally, the identification module 740 is specifically used for:

[0186] Determining a pixel category of each pixel on the receptive field image based on a pre-trained semantic segmentation model;

[0187] According to the pixel category of each pixel point in the wheel track area, the road surface type corresponding to the wheel track area is determined.

[0188] As the above device, optionally, further comprises a first training module, wherein the first training module is used to:

[0189] Obtain sample receptive field images, and mark the pixel category of each pixel point according to the road surface type of each pixel point in each sample receptive field image to obtain a receptive field image sample set;

[0190] Dividing the receptive field image sample set into a receptive field image training set and a receptive field image test set;

[0191] Build a semantic segmentation model based on a lightweight backbone network;

[0192] Input each sample receptive field image in the receptive field image training set into the semantic segmentation model, perform multi-scale semantic feature extraction on the sample receptive field image through the semantic segmentation model, and output the pixel category of each pixel point;

[0193] Based on the receptive field image test set, the semantic segmentation model is optimized.

[0194] As in the above device, optionally, the identification module 740 is specifically used for:

[0195] Traversing the pixel categories of each row of pixels in the wheel track area by column;

[0196] The number of pixels in each pixel category is counted row by row, and the pixel category weight is determined according to the total number of pixels in the row to obtain the normalized number of pixels in each pixel category;

[0197] After the traversal is completed, the normalized number of each pixel category in the wheel track area is sorted, and the pixel category with the largest normalized number is used as the road surface type corresponding to the wheel track area.

[0198] As in the above device, optionally, the identification module 740 is specifically used for:

[0199] Preprocessing the wheel track area into a wheel track rectangular area;

[0200] Inputting the wheel track rectangular area into a pre-trained multi-label classification network to determine the confidence of each road surface type in the wheel track rectangular area;

[0201] The road surface type corresponding to the wheel track area is determined according to the confidence level.

[0202] As the above device, optionally, further comprises a second training module, wherein the second training module is used to:

[0203] Obtaining sample wheel track rectangular images, marking multiple road surface types corresponding to each sample wheel track rectangular image, and obtaining a wheel track rectangular image sample set;

[0204] Dividing the wheel track rectangular image sample set into a wheel track rectangular image training set and a wheel track rectangular image test set;

[0205] Based on the deep convolutional neural network model, a multi-label classification network is constructed;

[0206] Inputting each sample wheel track rectangular image in the wheel track rectangular image training set into the multi-label classification network, performing multi-layer convolution through the multi-label classification network, and outputting the road surface type and corresponding confidence of each sample wheel track rectangular image;

[0207] Based on the wheel track rectangle image test set, the multi-label classification network is optimized.

[0208] As in the above device, optionally, the identification module 740 is further used for:

[0209] Determine the probability of each road surface type based on the confidence level of each road surface type;

[0210] Output each road surface type and the corresponding probability corresponding to the wheel track area.

[0211] As in the above device, optionally, the wheel track extraction module 730 is specifically used for:

[0212] Determine the vehicle's front direction and tire status based on the vehicle's current driving trajectory and predicted trajectory;

[0213] According to the vehicle head orientation and tire status, a left wheel track area and a right wheel track area are determined from the receptive field image.

[0214] As the above device, optionally, further comprises:

[0215] The sending module is used to send the road surface type corresponding to the wheel track area to the vehicle control device, so that the vehicle control device controls the vehicle according to the road surface type.

[0216] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment, which will not be repeated here.

[0217] In addition, an embodiment of the present invention further provides an autonomous driving vehicle, including a road surface type detection device as described in the above embodiment, which is used to implement a road surface type detection method. For details, see the above method embodiment. It should be noted that when implementing the present invention, model development involves python language, deep learning frameworks such as pytorch, tensorflow, etc., and when the road surface type detection method is deployed on the vehicle side, it involves c++ and model quantization tool related software; when the road surface type detection method is applied, hardware, sensors such as cameras, etc. are required for data collection, and hardware resources such as computing units such as central processing units CPU, graphics processing units GPU are required to support reasoning from data to road surface type detection methods; in addition, the output results of the road surface type detection method can be given to downstream modules, such as vehicle control devices, to achieve the best implementation effect.

[0218] Figure 8 FIG. 1 is a structural block diagram of an electronic device embodiment of the present invention, such as Figure 8 As shown, the device includes: a processor 810, a memory 820 and a bus 830;

[0219] The processor 810 and the memory 820 communicate with each other via the bus 830;

[0220] The processor 810 is used to call the program instructions in the memory 820 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0221] An embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: acquiring a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0222] An embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: acquiring a real-time image transmitted by a vehicle-mounted camera; preprocessing the real-time image to determine a receptive field image; determining a wheel track area from the receptive field image; and determining a road surface type corresponding to the wheel track area based on a pre-trained deep learning model.

[0223] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0224] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0225] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0226] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0229] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0231] The above is a detailed introduction to a road type detection method, a road type detection device, an automatic driving vehicle, an electronic device and a storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A road surface type detection method, characterized in that: include: Get real-time images transmitted by the vehicle camera; Preprocessing the real-time image to determine a receptive field image; Determining a wheel track area from the receptive field image, the wheel track area being an area of ​​the road surface where the wheel is about to travel; Determining the road surface type corresponding to the wheel track area according to a pre-trained deep learning model; Wherein, determining the wheel track area from the receptive field image includes: Determine the vehicle's front direction and tire status based on the vehicle's current driving trajectory and predicted trajectory; According to the vehicle head orientation and tire status, a left wheel track area and a right wheel track area are determined from the receptive field image.

2. The method according to claim 1, characterized in that Determining the road surface type corresponding to the wheel track area according to the pre-trained deep learning model includes: Determining a pixel category of each pixel on the receptive field image based on a pre-trained semantic segmentation model; According to the pixel category of each pixel point in the wheel track area, the road surface type corresponding to the wheel track area is determined.

3. The method according to claim 2, characterized in that The semantic segmentation model is trained by the following steps: Obtain sample receptive field images, and mark the pixel category of each pixel point according to the road surface type of each pixel point in each sample receptive field image to obtain a receptive field image sample set; Dividing the receptive field image sample set into a receptive field image training set and a receptive field image test set; Build a semantic segmentation model based on a lightweight backbone network; Input each sample receptive field image in the receptive field image training set into the semantic segmentation model, perform multi-scale semantic feature extraction on the sample receptive field image through the semantic segmentation model, and output the pixel category of each pixel point; Based on the receptive field image test set, the semantic segmentation model is optimized.

4. The method according to claim 3, characterized in that The determining, according to the pixel category of each pixel point in the wheel track area, the road surface type corresponding to the wheel track area includes: Traversing the pixel categories of each row of pixels in the wheel track area by column; Count the number of pixels in each pixel category by row, determine the weight of each pixel category according to the total number of pixels in each row, and obtain the normalized number of each pixel category; After the traversal is completed, the normalized number of each pixel category in the wheel track area is sorted, and the pixel category with the largest normalized number is used as the road surface type corresponding to the wheel track area.

5. The method according to claim 1, characterized in that Determining the road surface type corresponding to the wheel track area according to the pre-trained deep learning model includes: Preprocessing the wheel track area into a wheel track rectangular area; Inputting the wheel track rectangular area into a pre-trained multi-label classification network to determine the confidence of each road surface type in the wheel track rectangular area; The road surface type corresponding to the wheel track area is determined according to the confidence level.

6. The method according to claim 5, characterized in that The multi-label classification network is trained by the following steps: Obtaining sample wheel track rectangular images, marking multiple road surface types corresponding to each sample wheel track rectangular image, and obtaining a wheel track rectangular image sample set; Dividing the wheel track rectangular image sample set into a wheel track rectangular image training set and a wheel track rectangular image test set; Based on the deep convolutional neural network model, a multi-label classification network is constructed; Inputting each sample wheel track rectangular image in the wheel track rectangular image training set into the multi-label classification network, performing multi-layer convolution through the multi-label classification network, and outputting the road surface type and corresponding confidence of each sample wheel track rectangular image; Based on the wheel track rectangle image test set, the multi-label classification network is optimized.

7. The method according to claim 6, characterized in that After determining the confidence level of each road surface type in the wheel track rectangular area, the method further includes: Determine the probability of each road surface type based on the confidence level of each road surface type; Output each road surface type and the corresponding probability corresponding to the wheel track area.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The road surface type corresponding to the wheel track area is sent to a vehicle control device so that the vehicle control device controls the vehicle according to the road surface type.

9. A road surface type detection device, characterized in that: include: An acquisition module is used to acquire real-time images transmitted by the vehicle-mounted camera; A preprocessing module, used to preprocess the real-time image to determine a receptive field image; A wheel track extraction module, used to determine a wheel track area from the receptive field image, wherein the wheel track area is an area of ​​the road surface where the wheels are about to travel; An identification module, used to determine the road surface type corresponding to the wheel track area according to a pre-trained deep learning model; Wherein, the wheel track extraction module is specifically used for: Determine the vehicle's front direction and tire status based on the vehicle's current driving trajectory and predicted trajectory; According to the vehicle head orientation and tire status, a left wheel track area and a right wheel track area are determined from the receptive field image.

10. An autonomous driving vehicle, characterized in that: include: A road surface type detection device, wherein the road surface type detection device is used to execute the road surface type detection method as described in any one of claims 1-8.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor and the memory communicate with each other via a bus; The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images