Road adhesion coefficient estimation method, device, equipment and related program product
By performing semantic segmentation and classification on video stream data, the road surface type of the region of interest is identified, which solves the problem of poor accuracy of traditional road surface adhesion coefficient estimation methods under low excitation conditions. This enables accurate estimation and flexible response of the road surface adhesion coefficient, improving vehicle driving safety and control precision.
Patent Information
- Application Number
- CN202510859143.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2025-11-14
AI Technical Summary
Traditional methods for estimating road surface adhesion coefficients are inaccurate under low-excitation conditions and lack the ability to predict sudden changes in road surface conditions, making it difficult to meet the accuracy and safety requirements of vehicle dynamic control.
By acquiring video stream data, extracting multiple frames of images and performing semantic segmentation, identifying the road surface type of the region of interest, estimating the adhesion coefficient using semantic segmentation feature maps and a road surface image classification network model, and predicting the driving route by combining vehicle position and steering wheel angle information, an accurate estimation of the road surface adhesion coefficient is achieved.
It improves the accuracy and flexibility of road surface adhesion coefficient estimation, enabling it to cope with various sudden changes in road conditions and enhance vehicle driving safety and control precision.
Smart Images
Figure CN120953649A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle technology, and in particular to a method, apparatus, equipment and related program products for predicting road surface adhesion coefficient. Background Technology
[0002] Traditional methods for estimating the coefficient of friction (COP) are primarily effect-based. These methods infer the COP by analyzing the vehicle's dynamic response under various road conditions. This approach relies heavily on observing the impact of road excitation on vehicle dynamics, particularly tire slip ratio and the interaction force between the tire and the road surface. While effect-based methods often yield relatively accurate COPs under specific conditions with sufficient excitation (such as large vehicle acceleration or large tire forces), they also exhibit some hysteresis. When road excitation is low, the estimation results from effect-based methods often show strong randomness, reducing their reliability. More importantly, because effect-based methods estimate the COP based on the current dynamic response of the tires and vehicle body, they have poor predictability regarding the road surface adhesion ahead and limited responsiveness to sudden changes in road conditions. In summary, traditional effect-based COP estimation methods suffer from estimation hysteresis due to reliance on strong excitation conditions, strong randomness under low excitation conditions, and a lack of predictability for sudden changes in road surface adhesion ahead, making it difficult to meet the accuracy and safety requirements of vehicle dynamic control. Summary of the Invention
[0003] This application provides a method, apparatus, equipment, and related program products for estimating road surface adhesion coefficient, which can solve the problem of inaccurate adhesion coefficient estimation in traditional methods.
[0004] According to one aspect of the embodiments of this application, a method for predicting the road surface adhesion coefficient is proposed, the method comprising: Acquire video stream data for the road area; Extract multiple frames of images from the video stream data; Determine the semantic segmentation feature maps corresponding to the multiple frames of images, wherein the semantic segmentation feature maps include road surface regions; For each semantic segmentation feature map, multiple regions of interest are extracted from the road surface region in the semantic segmentation feature map, the road surface type identification result of each region of interest is determined, and the road surface type corresponding to the semantic segmentation feature map is determined based on the road surface type identification result of each region of interest. The set of road surface adhesion coefficients for the road region is determined based on the road surface type corresponding to each of the semantic segmentation feature maps.
[0005] In the above scheme, determining the semantic segmentation feature maps corresponding to the multiple frames of images respectively includes: The multi-frame images are input into a preset semantic segmentation network model to obtain semantic segmentation feature maps that correspond one-to-one with the multi-frame images.
[0006] In the above scheme, the preset semantic segmentation network model is trained through the following steps: Determine the initial semantic segmentation network model; Construct a sample training set for the initial semantic segmentation network model, the sample training set including semantic segmentation data of multiple labeled road surface regions; The initial semantic segmentation network model is iteratively trained based on each semantic segmentation data in the sample training set until the cross-entropy loss function of the trained preset semantic segmentation network model is lower than a preset threshold.
[0007] In the above scheme, the extraction of multiple regions of interest from the road surface region in the semantic segmentation feature map includes: Determine the vehicle's position, speed, and steering wheel angle information; The predicted driving route of the vehicle is determined based on the location, the vehicle speed, and the steering wheel angle information; Based on the predicted driving route, a first region of interest is determined in the road surface area to characterize the vehicle turning left, a second region of interest to characterize the vehicle going straight, and a third region of interest to characterize the vehicle turning right.
[0008] In the above scheme, determining the road surface type identification result for each of the regions of interest includes: The first region of interest, the second region of interest, and the third region of interest are respectively input into a preset road image classification network model to obtain a first recognition result corresponding to the first region of interest and the confidence level of the first recognition result, a second recognition result corresponding to the second region of interest and the confidence level of the second recognition result, and a third recognition result corresponding to the third region of interest and the confidence level of the third recognition result.
[0009] In the above scheme, determining the road surface type corresponding to the semantic segmentation feature map based on the road surface type identification results of each of the regions of interest includes: The road surface type corresponding to the semantic segmentation feature map is determined based on the first identification result and its confidence level, the second identification result and its confidence level, and the third identification result and its confidence level.
[0010] In the above scheme, determining the set of road surface adhesion coefficients for the road region based on the road surface type corresponding to each of the semantic segmentation feature maps includes: For each semantic segmentation feature map, the road surface type corresponding to the semantic segmentation feature map is input into a preset mapping table to obtain the adhesion coefficient of the vehicle in the road surface area of the semantic segmentation feature map. The set of road surface adhesion coefficients for the road region is determined based on the adhesion coefficients of the vehicle in the road surface regions of each of the semantic segmentation feature maps.
[0011] According to one aspect of the embodiments of this application, a road surface adhesion coefficient prediction device is provided, the device comprising: The acquisition unit is used to acquire video stream data for the road area; The extraction unit is used to extract multiple frames of images from the video stream data; The first determining unit is used to determine the semantic segmentation feature maps corresponding to the multiple frames of images, wherein the semantic segmentation feature maps include road surface regions. The second determining unit is used to extract multiple regions of interest from the road surface region in each semantic segmentation feature map, determine the road surface type identification result of each region of interest, and determine the road surface type corresponding to the semantic segmentation feature map based on the road surface type identification result of each region of interest. The third determining unit is used to determine the set of road surface adhesion coefficients of the road area according to the road surface type corresponding to each of the semantic segmentation feature maps.
[0012] According to one aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the road surface adhesion coefficient prediction method as described above. According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being read and executed by a processor of an electronic device, causing the electronic device to perform the road surface adhesion coefficient prediction method as described above.
[0013] The beneficial effects of this application are as follows: By acquiring video stream data of the road area, specifically through a camera device mounted on a vehicle, and then extracting multiple frames from the video stream data, the semantic segmentation feature map corresponding to each frame is determined. This semantic segmentation feature map includes the road surface area. Therefore, multiple regions of interest (ROIs) are extracted from the road surface area of the semantic segmentation feature map, and the road surface type identification results of each ROI are determined. The road surface type identification results of each ROI are comprehensively considered to obtain the road surface type corresponding to the semantic segmentation feature map. This application estimates the adhesion coefficient through visual analysis, specifically through a video stream-image-semantic segmentation feature map approach. This avoids the problem of inaccurate adhesion coefficient prediction caused by interference when road surface excitation is small, and can flexibly handle various abrupt changes in road surface conditions, thereby accurately predicting the changes in the adhesion coefficient (i.e., the adhesion coefficient set) of the vehicle traveling in the road area. Therefore, this application can solve the problem of inaccurate adhesion coefficient estimation in traditional road surface adhesion coefficient estimation methods, and improve the accuracy of adhesion coefficient prediction. Attached Figure Description
[0014] Figure 1 This is a system architecture diagram of the road surface adhesion coefficient prediction method provided in the embodiments of this application; Figure 2 A schematic flowchart illustrating the method for estimating the road surface adhesion coefficient provided in this application embodiment; Figure 3 A schematic diagram of the overall logic of the road surface adhesion coefficient prediction method provided in the embodiments of this application; Figure 4 A schematic diagram of the region of interest provided in the embodiments of this application; Figure 5 A schematic diagram of a semantic segmentation feature map provided in an embodiment of this application; Figure 6 A block diagram of the road surface adhesion coefficient prediction device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation
[0015] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0016] It should be noted that while some processes described in the specification, claims, and accompanying drawings include multiple steps appearing in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not themselves represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" in this document refers to at least two.
[0017] It is worth noting that in the specific embodiments of this application, video, image, and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target object is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain video or image data, separate permission or consent from the target object can be obtained through pop-up windows or redirection to a confirmation page. After obtaining the separate permission or consent from the target object, the necessary video and image data for enabling the embodiment of this application to operate normally can then be obtained.
[0018] Please see Figure 1 , Figure 1 This is a system architecture diagram of the road surface adhesion coefficient prediction method provided in this application embodiment. It includes a terminal 140, an Internet connection 130, a gateway 120, a server 110, etc.
[0019] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.
[0020] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.
[0021] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.
[0022] The following provides a detailed description of the specific implementation methods of the embodiments of this application: Please see Figure 2 , Figure 2 This is a flowchart illustrating the method for estimating the road surface adhesion coefficient provided in this application embodiment. The method for estimating the road surface adhesion coefficient can be implemented by server 110 and / or terminal 140. Figure 2 The methods for predicting the road surface adhesion coefficient shown include: Step 210: Obtain video stream data for the road area; Step 220: Extract multiple frames of images from the video stream data; Step 230: Determine the semantic segmentation feature maps corresponding to the multiple frames of images, wherein the semantic segmentation feature maps include road surface regions; Step 240: For each semantic segmentation feature map, extract multiple regions of interest from the road surface region in the semantic segmentation feature map, determine the road surface type identification result of each region of interest, and determine the road surface type corresponding to the semantic segmentation feature map based on the road surface type identification result of each region of interest. Step 250: Determine the set of road surface adhesion coefficients for the road area based on the road surface type corresponding to each of the semantic segmentation feature maps.
[0023] In step 210, the road area can be roads within a preset range. The preset range can be set according to actual needs. For example, the road area can be the roads between sections XX and XX of a national highway, which is the preset range. Similarly, the road area between XX Road and XX Road in a city can also be considered as the road area within the preset range. Video stream data of the road area can be acquired using camera devices installed on the vehicle. For example, a dashcam can record the road area the vehicle travels through to obtain video stream data. Of course, the acquisition method of video stream data is not limited to dashcams; it can also be acquired through other camera devices. Therefore, the acquisition method of video stream is not limited here.
[0024] In step 220, single or multiple frames of images can be extracted from the video stream data at a fixed frame rate, that is, the video stream data is divided into images. The images can be analyzed frame by frame in a detailed and accurate manner, thereby improving the accuracy of the adhesion coefficient prediction for the road area.
[0025] In step 230, the extracted multi-frame images can be input into a preset semantic segmentation network model, and the semantic segmentation feature maps corresponding to the multi-frame images can be determined by the preset semantic segmentation network model. For example Figure 3 As shown, video stream data acquired by a vehicle-mounted front-facing camera can be converted into multiple single-frame images. These single-frame images are then input into a pre-defined semantic segmentation network model for semantic segmentation, resulting in road surface and non-road surface regions. Finally, a semantic segmentation feature map is obtained, which includes both road surface and non-road surface regions. The pre-defined semantic segmentation network model can extract the road surface region from the image while masking non-road surface regions, noise, and other parts unrelated to the road surface. Figure 3 The semantic segmentation feature map clearly shows that only the road surface area in the road region is retained, while other factors unrelated to the road surface area are removed.
[0026] It should be noted that the preset semantic segmentation network model can be based on the lightweight Deeplabv3+. Deeplabv3+ is a semantic segmentation model that significantly improves segmentation accuracy by introducing an encoder-decoder structure and an improved atrous convolution design. Applied to this scenario, it performs excellently in processing lane and road edge details, thus accurately extracting the contour map of the road surface region in the current image. The semantic segmentation feature map output by the semantic segmentation network model is a binary image. Road surface regions are marked as 1, and non-road surface regions are marked as 0. A pixel-by-pixel logical operation is performed between this binary image and the original image. In this way, only the pixels corresponding to the road surface regions marked as 1 in the binary image are retained, while pixels at other locations are filtered out. The specific segmentation and road surface extraction results are illustrated below. Figure 5 As shown.
[0027] In step 240, multiple regions of interest (ROIs) are extracted from the semantic segmentation feature map to determine the road surface type identification result for each ROI, and the road surface type corresponding to the semantic segmentation feature map is determined based on the road surface type identification result of each ROI. (Continue referring to...) Figure 3 As shown, the semantic segmentation result (semantic segmentation feature map) is used to extract the region of interest.
[0028] In some embodiments, extracting multiple regions of interest from the road surface region in the semantic segmentation feature map includes: Determine the vehicle's position, speed, and steering wheel angle information; The predicted driving route of the vehicle is determined based on the location, the vehicle speed, and the steering wheel angle information; Based on the predicted driving route, a first region of interest is determined in the road surface area to characterize the vehicle turning left, a second region of interest to characterize the vehicle going straight, and a third region of interest to characterize the vehicle turning right.
[0029] Specific diagrams are as follows Figure 4 As shown, Figure 4The two green trajectory lines represent the predicted driving route of the vehicle, while the red dashed box represents the first region of interest, the yellow dashed box represents the second region of interest, and the blue dashed box represents the third region of interest. The vehicle's current position can be determined by analyzing the image and the position of the cameras on the vehicle, or by using a position sensor. Similarly, vehicle speed and steering wheel angle information can be obtained through sensors. After obtaining the position, speed, and steering wheel angle information, the vehicle's driving route can be predicted, as shown below. Figure 4 The green trajectory line shown.
[0030] In some embodiments, the predicted driving route of the vehicle can be determined based on the vehicle's length and width dimensions, wheel position information, wheel size information, combined with vehicle speed and steering wheel angle information. For example, if the wheel width is larger, the predicted driving route width will be larger, and if the wheel width is smaller, the predicted driving route width will also be smaller. This can more accurately determine the first region of interest, the second region of interest, and the third region of interest, facilitating better subsequent estimation of the adhesion coefficient.
[0031] Before inputting the region of interest into the preset road image classification network model, a judgment needs to be made. That is, if there are non-road areas in the region of interest, the proportion of the area of the non-road areas in the region of interest is determined. If the proportion of the area is more than 30%, then the region of interest will not be input into the preset road image classification network model for recognition, so as to avoid significant recognition errors caused by background noise.
[0032] In some embodiments, determining the road surface type identification result for each of the regions of interest includes: The first region of interest, the second region of interest, and the third region of interest are respectively input into a preset road image classification network model to obtain a first recognition result corresponding to the first region of interest and the confidence level of the first recognition result, a second recognition result corresponding to the second region of interest and the confidence level of the second recognition result, and a third recognition result corresponding to the third region of interest and the confidence level of the third recognition result.
[0033] Specifically, the road surface type corresponding to the semantic segmentation feature map can be determined based on the first identification result and its confidence level, the second identification result and its confidence level, and the third identification result and its confidence level. The trained road surface image classification network model will simultaneously identify the images corresponding to the six extracted regions of interest (ROIs). Each ROI corresponds to one identification result; that is, two first ROIs correspond to two first identification results, two second ROIs correspond to two second identification results, and two third ROIs correspond to two third identification results. Therefore, there are six identification results for each of the six ROIs, and each identification result corresponds to a confidence level.
[0034] The identification results will output a road surface type and its corresponding confidence score. First, there are multiple road surface types, for example, 28 common types, named in the format of friction level_road material_road roughness. The friction level attribute includes six subcategories corresponding to different weather conditions: dry, wet, water, fresh snow, melted snow, and ice. Road material attributes include asphalt, concrete, mud, and gravel. Road roughness is categorized according to the degree of road surface damage: smooth, minor, and severe. These 28 common road surface types specifically include: Fresh Snow, Water-Clay, Water-Gravel, Heavy Water-Asphalt, Slight Water-Asphalt, Smooth Water-Asphalt, Heavy Water-Concrete, Slight Water-Concrete, Smooth Water-Concrete, Melted Snow, Dry Mud, Dry Gravel, Heavy Dry Asphalt, Slight Dry Asphalt, Smooth Dry Asphalt, Heavy Dry Concrete, Slight Dry Concrete, Smooth Dry Concrete, Damp Mud, Damp Gravel, Heavy Damp Damp Asphalt, Slight Damp Damp Asphalt, Smooth Damp Damp Concrete, Smooth Damp Concrete, Smooth Damp Concrete, Dry Ice, Damp Ice.
[0035] The final pavement type is determined using a majority voting method based on the six identification results. That is, the pavement type with the most frequent identifications is selected as the final pavement type. For example, if three of the six identification results are "water-asphalt-slight," two are "water-asphalt-smooth," and one is "water-asphalt-heavy," then the pavement type corresponding to the three consistent identification results, i.e., "water-asphalt-slight," can be selected as the final pavement type. Another scenario is where the pavement types corresponding to the six identification results are inconsistent. In this case, the pavement type is determined by the confidence level of the pavement types corresponding to the six identification results, and the pavement type corresponding to the identification result with the highest confidence level is selected as the final pavement type.
[0036] The pre-defined semantic segmentation network model is trained through the following steps: Determine the initial semantic segmentation network model; Construct a sample training set for the initial semantic segmentation network model, the sample training set including semantic segmentation data of multiple labeled road surface regions; The initial semantic segmentation network model is iteratively trained based on each semantic segmentation data in the sample training set until the cross-entropy loss function of the trained preset semantic segmentation network model is lower than a preset threshold.
[0037] For semantic segmentation tasks involving road surface areas, the images to be semantically segmented (i.e., the multi-frame images extracted from video stream data as described in this application) often contain complex surrounding natural environments and traffic conditions, which greatly increases the difficulty of semantic segmentation. To obtain a more accurate semantic segmentation network model, this application utilizes a large-scale set of image samples (i.e., the semantic segmentation data described in this application) for training the semantic segmentation network model. During the construction of the sample training set, semantic categories need to be predefined, and annotation software is used to perform high-precision annotation on each image sample (semantic segmentation data) in the sample training set.
[0038] Based on this, this application selects an initial semantic segmentation network model, or a deep learning model, to construct a sample training set for the initial semantic segmentation network model. The sample training set for the initial semantic segmentation network model consists of image samples extracted frame by frame from video stream data corresponding to 28 common road surface types (the number of image samples is not limited, but the sample training set must contain image samples corresponding to 28 common road surface types). Each image sample corresponding to a road surface type has been labeled with a road surface region. This is used to train the initial semantic segmentation network model. In this way, the pre-set semantic segmentation network model obtained by training can accurately extract the road surface region in the image.
[0039] For example Figure 5 As shown, semantic segmentation is performed on the original image of the road surface type dry_asphalt_smooth. The original image is obtained by extracting video stream data at a fixed frame rate as described above. Semantic segmentation of the original image yields the following results: Figure 5 The semantic segmentation result shown is obtained through binarization, where pixels in the red area correspond to 1 and pixels in the black area correspond to 0. Finally, the road surface area of the original image is preserved to generate a semantic segmentation feature map.
[0040] Furthermore, the road surface image classification network model is trained in the following way: First, the image dataset consisting of images of 28 road surface types is labeled with road surface types. For example, an image of a road surface that has just been snowed on is labeled as the "fresh snow" road surface type. Similarly, all images in the image dataset are labeled with road surface types. Then, the image dataset is used for training, and data augmentation is combined to complete the training of the road surface image classification network model, thus obtaining the final road surface image classification network model.
[0041] The data augmentation technique introduced during the training phase of the road classification network model involves randomly shifting the input image and randomly modifying its brightness, contrast, saturation, and hue to increase the diversity of samples. This enhances the generalization ability of the road classification network model, thereby improving its adaptability and performance under different environmental conditions.
[0042] The various data augmentation methods are as follows: (1) Random translation: Before inputting the image into the model for training, the image is randomly translated, with the maximum translation amount in the X and Y directions being 15% of the image size. In road surface type recognition, this helps the road surface classification network model resist interference from non-road surface areas caused by region of interest clipping.
[0043] (2) Brightness Adjustment: The image brightness will be increased or decreased by a random value of up to 0.3 times the original brightness. The brightness of the road surface may vary greatly depending on different times and weather conditions. This adjustment allows the road surface classification network model to better adapt to different lighting conditions.
[0044] (3) Contrast Adjustment: During data augmentation, the contrast of each image is randomly adjusted to a value between 0.7 and 1.3 times. This adjustment helps to change the visual intensity of road surface texture and shape features, making them either more prominent or slightly blurred in the image. In this way, the road surface classification network model can more effectively identify and understand road surface features under different contrast environments, thereby improving its accuracy in identifying complex road surface conditions.
[0045] (4) Tone Adjustment: The tone of the image is randomly adjusted, with the range of variation limited to ±20% of the angle. Tone adjustment can simulate the color shift of the image under different lighting conditions, thereby enabling the road classification network model to adapt to images captured under different lighting conditions, which helps to improve the adaptability and recognition accuracy of the road classification network model to changes in light.
[0046] During training, the cross-entropy loss function is chosen.
[0047] In the formula This indicates the number of samples, corresponding to the number of images in multiple frames. Indicates the number of road surface types. y ij For the label function (when sample (The value is 1 if the true label is equal to the j-th category, and 0 otherwise.) p ij Samples predicted by the road classification network model Belongs to the The probability of a category after normalization using the softmax function.
[0048] The pre-defined road image classification network model is MobileNetV3 Large, which is used for road type recognition. Before training begins, the weights of the MobileNetV3 model, pre-trained based on the image dataset, are loaded. To better handle noise and uncertainty in the training data and improve the robustness and generalization ability of the road image classification network model, a label smoothing algorithm is introduced based on the cross-entropy loss function. Label smoothing is achieved by applying the true discrete label function when calculating the cross-entropy loss. Replace with a smooth probability distribution Specifically, a smoothing parameter is subtracted from each value in the actual label, and the remaining portion is then evenly distributed among the other items, as shown in the following formula:
[0049] in In this embodiment of the application, 0.1 is used. This represents the number of road surface types. Introducing the label smoothing strategy into the cross-entropy loss function yields:
[0050] The overall logic of the embodiments of this application is as follows: Figure 3 As shown, real-time video stream data from the vehicle's front captured by an onboard camera is cropped into images at a specific frame rate. A trained semantic segmentation network model is used to extract road surface features while masking non-road surface areas, resulting in a semantic segmentation feature map. Regions of interest (ROIs) are then extracted, and a trained road surface image classification network model accurately classifies the ROI images. The recognition results of the ROI images are then combined and output using a majority voting method. Finally, based on the mapping relationship between road surface type, vehicle speed, and road surface adhesion coefficient, an estimated value of the road surface adhesion coefficient in front of the vehicle is obtained.
[0051] In some embodiments, for each semantic segmentation feature map, the road surface type corresponding to the semantic segmentation feature map is input into a preset mapping table to obtain the adhesion coefficient of the vehicle in the road surface area of the semantic segmentation feature map. The set of road surface adhesion coefficients for the road region is determined based on the adhesion coefficients of the vehicle in the road surface regions of each of the semantic segmentation feature maps.
[0052] Specifically, to achieve the mapping and conversion between road surface type, vehicle speed, and coefficient of adhesion, the coefficient of adhesion is estimated using Tables 1 and 2 below. First, to fully consider the differences in coefficient of adhesion between high and low vehicle speeds on different road surfaces, the corresponding coefficient of adhesion mapping values for some road surface types (such as the 20 road surface types in Table 1) are set as the average of the upper and lower boundaries of the corresponding coefficient of adhesion, calculated according to the following formula:
[0053] In the formula Table 2 shows the upper and lower boundary values for the adhesion coefficient corresponding to different road surface types. For example, when the vehicle speed is below 48 km / h, and the vehicle is traveling on a dry, smooth concrete road surface, the upper boundary value is 1.0 and the lower boundary value is 0.8. Therefore, the final estimated adhesion coefficient is 0.9 (see Table 2). The upper and lower boundary values include both the upper and lower boundary values. This is the final adhesion coefficient. Therefore, Table 2 shows the adhesion coefficients corresponding to the 20 road surface types in Table 1, calculated using the methods described above.
[0054]
[0055] Table 1 Referring to the example in Table 1, which shows the upper and lower boundary values for each of the 20 road surface types, the mapping relationship between road surface type, vehicle speed, and adhesion coefficient can be further determined by combining Table 2.
[0056]
[0057] Table 2 Table 2 shows the estimated adhesion coefficient values obtained by averaging the upper and lower boundary values from Table 1. Table 2 also includes eight adhesion coefficients related to water, thus forming estimated adhesion coefficient values for 28 road surface types. Since relevant regulations do not specify the range of adhesion coefficients for wet road surfaces, this application roughly uses the lower boundary value of the adhesion coefficient corresponding to the high-speed driving of a vehicle on a wet road surface. Mapping is performed. To improve overall safety, the lower boundary value of the adhesion coefficient for icy surfaces is taken. For the coefficient of adhesion on snow surfaces, take the lower boundary value within the range. Based on the above, the coefficient values were appropriately lowered according to the increase in speed and the change in road surface moisture, and the specific numerical mapping relationship between road surface type and adhesion coefficient was obtained as shown in Table 2 (that is, the eight adhesion coefficients in the water column).
[0058] Finally, in step 250, the set of road surface adhesion coefficients for the road area can be determined based on the adhesion coefficients of the vehicle in the road surface areas corresponding to each of the semantic segmentation feature maps. That is, during vehicle operation, the adhesion coefficients of the road ahead are continuously estimated, forming a set of road surface adhesion coefficients. This set of adhesion coefficients allows for the determination of the road surface adhesion coefficients corresponding to different positions of the vehicle within the road area. Accurate estimation of the adhesion coefficients can effectively assist in driving, perform vehicle state analysis, and improve driving safety.
[0059] Please see Figure 6 , Figure 6 This is a schematic diagram of the road surface adhesion coefficient prediction device provided in an embodiment of this application. The road surface adhesion coefficient prediction device is applied to computer equipment, and the road surface adhesion coefficient prediction may include: Acquisition unit 601 is used to acquire video stream data for the road area; Extraction unit 602 is used to extract multiple frames of images from the video stream data; The first determining unit 603 is used to determine the semantic segmentation feature maps corresponding to the multiple frames of images respectively; The second determining unit 604 is used to extract multiple regions of interest from each semantic segmentation feature map, determine the road surface type recognition result of each region of interest, and determine the road surface type corresponding to the semantic segmentation feature map based on the road surface type recognition result of each region of interest. The third determining unit 605 is used to determine the set of road surface adhesion coefficients of the road area according to the road surface type corresponding to each of the semantic segmentation feature maps.
[0060] Reference Figure 7 , Figure 7 To implement the structural block diagram of a portion of the terminal 140 in this application embodiment, the terminal 140 includes: a radio frequency (RF) circuit 710, a memory 715, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790, among other components. Those skilled in the art will understand that... Figure 7The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0061] The RF circuit 710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 780; in addition, it transmits uplink data to the base station.
[0062] The memory 715 can be used to store software programs and modules. The processor 780 executes various terminal functions and road surface adhesion coefficient prediction processing by running the software programs and modules stored in the memory 715.
[0063] The input unit 730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732.
[0064] The display unit 740 can be used to display input or provided information, as well as various menus of the terminal. The display unit 740 may include a display panel 741.
[0065] Audio circuitry 760, speaker 761, and microphone 762 provide an audio interface.
[0066] In this embodiment, the processor 780 included in the terminal 140 can execute the road surface adhesion coefficient prediction method of the previous embodiment.
[0067] The terminal 140 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.
[0068] Figure 8This is a partial structural block diagram of a server 110 implementing an embodiment of this application. The server 110 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 110. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 110.
[0069] Server 110 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0070] The central processing unit 822 in server 110 can be used to execute the road surface adhesion coefficient prediction method of the embodiments of this application.
[0071] This application also provides a computer-readable storage medium for storing program code for executing the road surface adhesion coefficient prediction method of the foregoing embodiments.
[0072] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the above-described method for predicting the road surface adhesion coefficient.
[0073] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0074] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0075] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.
[0076] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.
[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0080] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.
[0081] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0082] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for predicting road surface adhesion coefficient, characterized in that, The method includes: Acquire video stream data for the road area; Extract multiple frames of images from the video stream data; Determine the semantic segmentation feature maps corresponding to the multiple frames of images, wherein the semantic segmentation feature maps include road surface regions; For each semantic segmentation feature map, multiple regions of interest are extracted from the road surface region in the semantic segmentation feature map, the road surface type identification result of each region of interest is determined, and the road surface type corresponding to the semantic segmentation feature map is determined based on the road surface type identification result of each region of interest. The set of road surface adhesion coefficients for the road region is determined based on the road surface type corresponding to each of the semantic segmentation feature maps.
2. The method for predicting the road surface adhesion coefficient according to claim 1, characterized in that, Determining the semantic segmentation feature maps corresponding to the multiple frames of images includes: The multi-frame images are input into a preset semantic segmentation network model to obtain semantic segmentation feature maps that correspond one-to-one with the multi-frame images.
3. The method for predicting the road surface adhesion coefficient according to claim 2, characterized in that, The preset semantic segmentation network model is trained through the following steps: Determine the initial semantic segmentation network model; Construct a sample training set for the initial semantic segmentation network model, the sample training set including semantic segmentation data of multiple labeled road surface regions; The initial semantic segmentation network model is iteratively trained based on each semantic segmentation data in the sample training set until the cross-entropy loss function of the trained preset semantic segmentation network model is lower than a preset threshold.
4. The method for predicting the road surface adhesion coefficient according to claim 1, characterized in that, The extraction of multiple regions of interest from the road surface region in the semantic segmentation feature map includes: Determine the vehicle's position, speed, and steering wheel angle information; The predicted driving route of the vehicle is determined based on the location, the vehicle speed, and the steering wheel angle information; Based on the predicted driving route, a first region of interest is determined in the road surface area to characterize the vehicle turning left, a second region of interest to characterize the vehicle going straight, and a third region of interest to characterize the vehicle turning right.
5. The method for predicting the road surface adhesion coefficient according to claim 4, characterized in that, The determination of the road surface type identification results for each of the regions of interest includes: The first region of interest, the second region of interest, and the third region of interest are respectively input into a preset road image classification network model to obtain a first recognition result corresponding to the first region of interest and the confidence level of the first recognition result, a second recognition result corresponding to the second region of interest and the confidence level of the second recognition result, and a third recognition result corresponding to the third region of interest and the confidence level of the third recognition result.
6. The method for predicting the road surface adhesion coefficient according to claim 5, characterized in that, The step of determining the road surface type corresponding to the semantic segmentation feature map based on the road surface type identification results of each of the regions of interest includes: The road surface type corresponding to the semantic segmentation feature map is determined based on the first identification result and its confidence level, the second identification result and its confidence level, and the third identification result and its confidence level.
7. The method for predicting the road surface adhesion coefficient according to claim 6, characterized in that, The step of determining the set of road surface adhesion coefficients for the road region based on the road surface type corresponding to each of the semantic segmentation feature maps includes: For each semantic segmentation feature map, the road surface type corresponding to the semantic segmentation feature map is input into a preset mapping table to obtain the adhesion coefficient of the vehicle in the road surface area of the semantic segmentation feature map. The set of road surface adhesion coefficients for the road region is determined based on the adhesion coefficients of the vehicle in the road surface regions of each of the semantic segmentation feature maps.
8. A road surface adhesion coefficient prediction device, characterized in that, The device includes: The acquisition unit is used to acquire video stream data for the road area; The extraction unit is used to extract multiple frames of images from the video stream data; The first determining unit is used to determine the semantic segmentation feature maps corresponding to the multiple frames of images, wherein the semantic segmentation feature maps include road surface regions. The second determining unit is used to extract multiple regions of interest from the road surface region in each semantic segmentation feature map, determine the road surface type identification result of each region of interest, and determine the road surface type corresponding to the semantic segmentation feature map based on the road surface type identification result of each region of interest. The third determining unit is used to determine the set of road surface adhesion coefficients of the road area according to the road surface type corresponding to each of the semantic segmentation feature maps.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the road surface adhesion coefficient prediction method according to any one of claims 1 to 7.
10. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program is read and executed by the processor of the electronic device, causing the electronic device to perform the road surface adhesion coefficient prediction method according to any one of claims 1 to 7.