Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the alignment and fusion of images of vehicle tires at a plurality of timestamps, and obtaining a first image, thereby preventing the condition that the road information is shielded by vehicles and pedestrians; and inputting the first image into an NC-SSD module of the first module for feature fusion processing, and a fourth function module of the first module for feature enhancement processing to obtain a second image, thereby realizing cross-modal feature interaction of multiple features, accurately capturing a local spatial relationship, improving image segmentation precision, and improving image segmentation accuracy. The efficiency and the accuracy of feature processing are improved; and finally, quickly and accurately identifying the road surface type according to the second image to obtain the road surface adhesion coefficient of the vehicle, thereby achieving the technical effect of improving the precision and practicability of road surface adhesion coefficient estimation.

Description

Technical Field

[0001] This application relates to the field of intelligent autonomous driving technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of autonomous driving and electric drive technologies, vehicle drive control systems are gradually shifting from passive response to active prediction. Traditional drive strategies mainly rely on onboard data such as wheel speed sensors and IMUs (Inertial Measurement Units) to adjust driving force in real time by monitoring wheel slip ratio. For example, ASR (Traction Control System) reduces motor torque output when wheel slippage is detected, but its control lag means that there is still a risk of power interruption or loss of control on low-traction surfaces such as ice, snow, and wet roads. In recent years, the mainstream methods for estimating tire-road adhesion coefficient have been divided into three categories: dynamic-based methods, vision-based methods, and methods that combine dynamics and vision.

[0003] The methods described above often fail to fully utilize tire information and stimuli, usually due to: limited perception dimensions: mainstream solutions use a single sensor, such as a camera or LiDAR, to achieve road surface recognition. Firstly, vision-based road classification systems, such as the VGG (Visual Geometry Group) convolutional neural network, can identify 5-8 typical road surfaces, including dry asphalt, snow and ice, and gravel. However, they rely on fixed datasets for training and struggle to adapt to new road conditions such as oil stains and mixed materials, and are easily affected by lighting and shadows, leading to misjudgments. Secondly, traditional drive anti-skid systems use a "slip-after-control" mode, where slippage occurs first, followed by torque adjustment. Experiments show that on roads with abrupt changes in adhesion coefficient, such as the asphalt-ice transition zone, the system response delay can reach 0.5-1.2 seconds, causing instantaneous vehicle instability. Thirdly, current strategies are mostly based on preset "road surface-adhesion coefficient" mapping tables, such as setting the adhesion coefficient for snow and ice to 0.15. However, the actual adhesion coefficient changes dynamically due to slope, tire temperature, and wear conditions, and fixed parameters result in insufficient control accuracy. Fourthly, the vehicle's braking, drive, and suspension systems are controlled independently, lacking road information sharing and multi-system coordination.

[0004] Therefore, improving the accuracy and practicality of estimating the coefficient of adhesion between vehicle tires and the road surface has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application propose a data processing method, apparatus, electronic device, and storage medium that can improve the accuracy and practicality of road surface adhesion coefficient estimation.

[0006] The following technical solution is adopted in this application.

[0007] In a first aspect, embodiments of this application provide a data processing method, the method comprising: A first image of the vehicle is acquired; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps; a second image is obtained by performing feature processing on the first type of information in the first image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires; based on the second image, the road surface type of the road where the vehicle is located is determined; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface; based on the road surface type, the road adhesion coefficient of the vehicle is determined.

[0008] In some embodiments, acquiring a first image of the vehicle includes: Acquire a first initial image of a vehicle with multiple timestamps; the first initial image includes a first type of information and a second type of information; the first type of information includes road surface, lane lines, and vehicle tires; the second type of information includes buildings and vegetation; process the first initial image to obtain a first target image, the first target image being used to indicate: determine the depth information of each pixel in the first initial image; stitch the first target image according to the chronological order to obtain the first image.

[0009] In some embodiments, performing feature processing on the first image to obtain the second image further includes: The first model is invoked, with the first image as input and the second image as output. The first model includes multiple sub-models executed sequentially. Each sub-model includes a first functional module for performing feature extraction, a second functional module for performing feature fusion, and a third functional module for performing nonlinear transformation.

[0010] In some embodiments, a first model is invoked, taking a first image as input and outputting a second image; the first model includes multiple sub-models executed sequentially, including: The first image is input into the first model, and multiple first intermediate features are output from multiple sub-models; feature enhancement processing is performed on each first intermediate feature to obtain the second image.

[0011] In some embodiments, a first image is input into a first model, and multiple first intermediate features are output from multiple sub-models, including: The first image is input into the first functional module of the sub-model of the first model, and multiple features are output. The multiple features include: a first feature obtained by feature mapping, and a second, third, and fourth feature obtained by feature mapping and nonlinear transformation. The multiple features are input into the second functional module of the sub-model, and a second intermediate feature is output. The second functional module includes an NC-SSD module, which is used to indicate: perform feature fusion processing on the multiple features. The second intermediate feature is input into the third functional module of the sub-model, and a first intermediate feature is output. The third functional module includes an FNN module, which is used to indicate: perform nonlinear transformation processing on the second intermediate feature.

[0012] In some embodiments, feature enhancement processing is performed on each first intermediate feature to obtain a second image, including: The first intermediate feature is used as input to the fourth functional module of the first model, and the second image is output; the fourth functional module is used to indicate: perform feature enhancement processing on the first intermediate feature.

[0013] In some embodiments, determining the road surface type of the road where the vehicle is located based on the second image includes: Image fusion processing is performed on multiple second images to obtain the road surface type of the road where the vehicle is located.

[0014] In some embodiments, determining the vehicle's road adhesion coefficient based on road surface type includes: The first threshold for the road surface adhesion coefficient is determined based on the vehicle's body mass and the water depth of the road surface. The road surface adhesion coefficient of the vehicle is then determined based on the first threshold, the vehicle's speed, and the road surface type.

[0015] According to a second aspect of the embodiments of this application, a data processing apparatus is provided, the apparatus comprising: The first acquisition module is used to acquire a first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps; the first processing module is used to perform feature processing on the first type of information in the first image to obtain a second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires; the second processing module is used to determine the road surface type of the road where the vehicle is located based on the second image; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface; the third processing module is used to determine the road surface adhesion coefficient of the vehicle based on the road surface type.

[0016] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the data processing method described above is implemented.

[0017] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor or electronic device, implement the above-described data processing method.

[0018] In this application's solution, firstly, a first image is obtained by fusing multiple frames of vehicle tire images across multiple consecutive time stamps. This avoids situations where the road surface is obscured by numerous vehicles and pedestrians in urban road scenes. Secondly, the first image is input into a first module to obtain a second image. For the first type of information in the first image, such as the road surface, lane lines, and vehicle tires, multiple features are obtained through feature mapping and nonlinear transformation. These features are then fused using the NC-SSD module, achieving cross-modal feature interaction. Furthermore, by enhancing the activation of relevant features and suppressing irrelevant features, a relatively large spatial context is captured with lower computational cost, resulting in more accurate capture of local spatial relationships and effective feature fusion. Finally, the road surface type is determined based on the second image. Since the second image is obtained through feature fusion and feature enhancement, combining it with the second image to determine the road surface type can more accurately identify the current road conditions of the vehicle. Then, the road surface adhesion coefficient is obtained based on the road surface type, vehicle speed, and vehicle noise quality, improving the accuracy and practicality of the road surface adhesion coefficient estimation.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 This is a schematic diagram of a data processing method provided in an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0023] Figure 3 This is a flowchart illustrating a method for acquiring a first image according to an embodiment of this application.

[0024] Figure 4 This is a flowchart illustrating a method for acquiring a second image provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram illustrating another process for acquiring a second image, provided in an embodiment of this application.

[0026] Figure 6 This is a flowchart illustrating a method for obtaining the road surface adhesion coefficient, provided in an embodiment of this application.

[0027] Figure 7 This is a flowchart illustrating a method for obtaining a first intermediate feature according to an embodiment of this application.

[0028] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0029] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0030] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through specific embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] In conventional technologies, 60% of the energy loss in electric vehicles stems from non-optimal drive control, and 23% of traffic accidents are related to low-traction road surfaces. On one hand, the real-time performance and accuracy of the tire-road adhesion coefficient cannot be guaranteed simultaneously. High-precision road surface detection requires extensive computation, such as 3D point cloud modeling, which is insufficient for the real-time control demands of high-speed driving. On the other hand, vehicle pre-control strategies require multi-sensor fusion, such as cameras, LiDAR, and millimeter-wave radar, but this increases hardware costs. Furthermore, current road surface detection standards are not standardized; for example, anti-skid testing uses only a single indicator like build depth or friction coefficient, resulting in poor adaptability across different scenarios. Therefore, there is a problem that cannot simultaneously satisfy the estimation accuracy and practicality of the tire-road adhesion coefficient.

[0033] The data processing method provided in this application is intended to solve the above-mentioned technical problems of the prior art.

[0034] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the data processing method execution body provided in this application embodiment includes a first module 101 and a second module 102.

[0036] In one optional implementation, the first module 101 and the second module 102 refer to software units or modules. The first module 101 is used to determine a first initial image of the vehicle, the vehicle's speed, and the vehicle's sound quality. The second module 102 is used to fuse multiple frames of images of the vehicle's tires at multiple time stamps to obtain a first image, and to perform feature fusion and feature enhancement processing on the first image to obtain a second image. Simultaneously, it determines the road type based on the second image, and then determines the road surface adhesion coefficient.

[0037] In another alternative implementation, the first module 101 and the second module 102 refer to hardware devices.

[0038] For example, the first module 101 may include, but is not limited to, vehicle sensors, cameras, etc.

[0039] For example, the second module 102 may include, but is not limited to, electronic devices with data processing capabilities such as computers, host computers, servers, or data centers.

[0040] Optionally, the first module 101 and the second module 102 can communicate via wired or wireless connections. Wired connections may include, but are not limited to, buses, fiber optic cables, or network cables. Wireless connections may include, for example, transmission control protocol / internet protocol (TCP / IP), wireless local area network (WLAN) protocols, and remote direct memory access (RDMA) over converged ethernet (RoCE) protocols.

[0041] The following is combined with Figure 1 The first module 101 and the second module 102 shown here are exemplary descriptions of the data processing method provided in the embodiments of this application: First, the first initial image of the vehicle, the vehicle's driving speed, and the vehicle sound quality determined by the first module 101 are sent to the second module 102 via wired or wireless communication. The second module 102 obtains a first image based on the first initial image, and performs feature fusion and feature enhancement processing on the first image to obtain a second image. At the same time, the road type is determined based on the second image, and the road surface adhesion coefficient is determined by combining the vehicle's driving speed and the vehicle sound quality.

[0042] Below Figure 1 Based on the first module 101 and the second module 102 shown, the data processing method provided in the embodiments of this application will be further described, such as... Figure 2 The diagram illustrates a data processing method. In a specific embodiment, this data processing method can be applied to, for example... Figure 8 The data processing device 800 and the electronic device 900 on which the data processing device 800 is arranged are shown. Figure 9 The specific process of the embodiments of this application will be described below. Of course, it is understood that this method can be executed by a cloud server with computing power. The following will focus on... Figure 2 The process shown is described in detail, and the interface processing method may specifically include the following steps 201 to 204.

[0043] Step 201: Obtain the first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps.

[0044] In this embodiment of the application, the first image is an image obtained by preprocessing the feature information in multiple frames of images and then stitching them together in chronological order.

[0045] For example, since images captured directly by a vehicle's camera contain a wide range of information, such as roads, buildings, sidewalks, the sky, or other vehicles, and also contain redundant information that can easily mislead the judgment of the type of road surface the vehicle is on, such as the color of the vehicle in front, it is necessary to preprocess the directly captured images in order to extract road-related information from the images more accurately.

[0046] For example, due to the diversity of vehicle scenarios, such as the presence of many other traffic participants around a vehicle in an urban setting, very little road-related information can be obtained from a single frame image, making it impossible for the vehicle to accurately identify its surroundings and pinpoint its location. Therefore, multiple frames with multiple timestamps can be obtained according to the vehicle's driving direction to increase the amount of road-related information on the road the vehicle is currently traveling on.

[0047] For example, during image preprocessing, the rotation trajectory of the wheels is determined based on the characteristics of the vehicle's left and right wheels, such as wheel hubs and tire treads, combined with the vehicle's driving direction. Then, by stitching together multiple frames from different timestamps, a relatively complete first image of the road surface where the vehicle is located can be obtained. Figure 4 The first image shown.

[0048] Step 202: Perform feature processing on the first type of information in the first image to obtain the second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires.

[0049] In this embodiment of the application, the first type of information is information used to identify the road surface type of the road where the vehicle is located, including information such as the road surface, lane lines, and vehicle tires in the first image.

[0050] In this embodiment of the application, the second image is an image used to determine the road surface type of the road where the vehicle is located.

[0051] In the first optional example, such as Figure 7 As shown, the first module includes multiple sub-models executed sequentially. These sub-models can adopt the VSSD (Visual State Space Duality) model. The first image is input into the VSSD model, and after feature mapping and nonlinear transformation, first feature A, second feature X, third feature B, and fourth feature C are obtained. These four features are then input into the NC-SSD module for feature fusion to obtain the second intermediate feature Y. After the second intermediate feature Y undergoes two residual summing processes, it is then processed by the FNN module for nonlinear transformation to obtain the first intermediate feature.

[0052] Optionally, the representational capabilities of the first intermediate features output by each sub-model are different. It can be understood that the first intermediate features output by the shallow sub-model can be used to represent features such as edges, corners, colors, and textures in the first image; the first intermediate features output by the middle sub-model can be used to represent parts of objects in the first image, such as vehicle tires; and the first intermediate features output by the high-level sub-model can be used to represent the entire object in the first image, such as a vehicle. By combining the first intermediate features with different representational capabilities, features with comprehensive spatial and semantic representation capabilities are obtained.

[0053] In the second alternative example, the first module constructed includes a fourth functional module, such as Figure 4The fourth functional module shown can employ the LGAG (Large-kernel Grouped Attention Gate) module. It uses 3×3 grouped convolutions to process the first intermediate features at each of the three levels. These first intermediate features are then merged element-wise, followed by an activation function to generate attention coefficients. These coefficients are then combined with the first intermediate features element-wise to obtain the attention-gated features, which are then output as shown below. Figure 4 The second image shown.

[0054] Step 203: Based on the second image, determine the road surface type of the road where the vehicle is located; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface.

[0055] In this embodiment of the application, the road surface type is classified in combination with the road surface and the road surface condition. Common road surfaces include asphalt road surface, gravel road surface, etc., and the road surface condition includes dry, wet, waterlogged, snow-covered, icy, etc.

[0056] For example, there is a mapping relationship between road surface type and the peak value of TRFC (Tire-Road Friction Coefficient), and the peak value of TRFC is constrained by a variety of factors, including contact materials, contact surface roughness, ambient temperature and humidity, etc.

[0057] For example, hydroplaning is a key factor that causes a sharp decrease in TRFC. When a vehicle travels at high speed over a flooded asphalt road, a water film forms on the contact surface between the tire and the road surface, causing the tire to temporarily separate from the road surface. This results in a complete loss of braking force, and the peak value of TRFC will decrease significantly. Therefore, when a vehicle is on a flooded asphalt road, the peak value of TRFC is mainly affected by changes in the vehicle's dynamic state and the depth of the water film, such as the vehicle's speed.

[0058] For example, the surface of snow and ice is mainly composed of solid or semi-solid water, which is difficult to form a liquid film that supports hydroplaning. Compared with waterlogged asphalt pavement, vehicles are less likely to hydroplan on dry asphalt pavement, semi-wet asphalt pavement, snow-covered pavement, and icy pavement. The peak value of TRFC is less affected by the dynamic state changes of the vehicle. Therefore, when vehicles are on dry asphalt pavement, semi-wet asphalt pavement, snow-covered pavement, and icy pavement, the peak value of TRFC is mainly affected by the material properties and micromorphology of the pavement itself, such as compacted snow and ice crystal structure.

[0059] In this embodiment of the application, the peak value of TRFC of the vehicle varies greatly on different road surface types. The second image obtained by feature processing the first image has clearer first-type information, which can accurately identify the road surface type of the road where the vehicle is located, thereby ensuring that the road surface adhesion coefficient of the vehicle can be accurately estimated.

[0060] Step 204: Determine the road adhesion coefficient of the vehicle based on the road surface type.

[0061] In this embodiment of the application, the road surface adhesion coefficient is the degree of adhesion ability of the vehicle's tires on different road surfaces.

[0062] For example, when a vehicle is on a wet or waterlogged asphalt road surface, hydroplaning can easily occur when the vehicle reaches a certain speed, causing a temporary loss of braking ability. The vehicle's road adhesion coefficient is related to the vehicle's mass, speed, and the depth of the water surface. When the vehicle's speed is below a first threshold, the road adhesion coefficient remains unchanged; when the vehicle's speed reaches the first threshold, the road adhesion coefficient decreases as the speed increases. The first threshold is positively correlated with the vehicle's mass and negatively correlated with the water depth.

[0063] For example, when a vehicle is on a muddy or post-snowmelt muddy road surface, field tests show that the vehicle's road adhesion coefficient is related to the driving speed. When the vehicle is traveling at a low speed, the road adhesion coefficient on a muddy or post-snowmelt muddy road surface is comparable to that on a road surface with medium adhesion; as the vehicle's speed increases, the road adhesion coefficient decreases significantly because a hydroplaning-like mud sliding layer is more likely to form on the muddy or post-snowmelt muddy road surface.

[0064] In this embodiment, a fusion estimation method is used to estimate the road adhesion coefficient of the vehicle in order to provide a key reference benchmark for the output of the image-based estimator (IBE) in fusion estimation frameworks such as autonomous driving or active safety systems.

[0065] For example, real-vehicle dynamic excitation tests were conducted on several representative road surfaces, including dry asphalt pavement, semi-wet asphalt pavement, waterlogged asphalt pavement, snow-covered pavement, rough icy pavement, and smooth icy pavement. These tests included step braking and cornering. All tests were conducted on a test vehicle with a vehicle weight of 2315 kg, and tire inflation pressure was maintained within the standard range (32-35 PSI, approximately 220-241 kPa). A large amount of test data was statistically analyzed and averaged to obtain the average peak TRFC for different road surface types, as shown in Table 1.

[0066] Table 1

[0067] For example, the road adhesion coefficient of a vehicle can be obtained by referring to Table 1 based on the vehicle's tire model and road surface type. Common tire specifications include 215 / 55R17, 225 / 60R19, and 245 / 45R19.

[0068] In this embodiment, firstly, a first image is obtained by stitching together multiple frames of images with multiple timestamps. Even though the road is heavily occluded in a single frame, the stitching process still yields a significant amount of road surface information. Secondly, multiple features obtained from processing the first type of information in the first image are input into the NC-SSD module for feature fusion to obtain second intermediate features. The NC-SSD module enhances the SSD blocks in visual applications, improving computational efficiency and accuracy. Furthermore, the second intermediate features are summed by residuals and then processed by the FNN module for nonlinear transformation, resulting in output features that incorporate multi-scale contextual information, achieving cross-modal feature interaction. The LGAG module further enhances the activation of relevant features and suppresses irrelevant features, improving the accuracy of features in image segmentation. Finally, the road surface type is determined based on the second image, and the road surface adhesion coefficient of the vehicle is determined by combining a first threshold and the vehicle's speed, thereby improving the accuracy and practicality of road surface adhesion coefficient estimation.

[0069] Regarding how to acquire the first image of the vehicle, embodiments of this application provide an optional implementation method, such as... Figure 3 The flowchart shown is a method for acquiring a first image, which may specifically include the following steps 301 to 307.

[0070] Step 301: Obtain a first initial image of the vehicle with multiple timestamps; the first initial image includes a first type of information and a second type of information; the first type of information includes road surface, lane lines, and vehicle tires; the second type of information includes buildings and vegetation.

[0071] In this embodiment of the application, the first initial image is an image directly captured by the vehicle's camera.

[0072] In this embodiment of the application, the second type of information is information unrelated to the road surface where the vehicle is located, including buildings, vegetation, sky, other traffic participants, etc. in the first image.

[0073] For example, during the vehicle's operation, images at different times are continuously captured by the vehicle's onboard camera in order to obtain more information about the road surface where the vehicle is located.

[0074] Step 302: Process the first initial image to obtain the first target image. The first target image is used to indicate: determine the depth information of each pixel in the first initial image.

[0075] In this embodiment of the application, the first target image is an image after differentiating and processing the first type of information and the second type of information in the first initial image.

[0076] For example, the first initial image captured by the camera is analyzed to separate drivable road areas such as road surface and lane lines from non-drivable road areas such as roadside vegetation and buildings, thus completing road segmentation of the image; based on the calculated distance relationship of each object in the image, for example, the brightness of the road surface texture and wheels in the foreground of the image is increased on the depth map, while the brightness of the trees and sidewalks in the distance of the image is decreased on the depth map, thus completing the depth mapping of the image.

[0077] Step 303: The first target image is stitched together according to the chronological order to obtain the first image.

[0078] For example, by combining the wheel hub and tire tread identified in the first target image, multiple frames of the first target image are stitched together in chronological order to obtain the first image. Compared with the case of directly capturing a single frame image with a camera, this can avoid the problem of vehicles and pedestrians obscuring road information in urban scenes.

[0079] Step 304: Obtain the first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps.

[0080] Step 305: Perform feature processing on the first type of information in the first image to obtain the second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires.

[0081] Step 306: Based on the second image, determine the road surface type of the road where the vehicle is located; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface.

[0082] Step 307: Determine the vehicle's road adhesion coefficient based on the road surface type.

[0083] The specific steps of steps 304 to 307 can be found in steps 201 to 204, and will not be repeated here.

[0084] In this embodiment of the application, by performing road segmentation and depth mapping preprocessing on the first initial image, the first type of information in the first image can be enhanced. Combined with dynamic alignment and fusion of images from consecutive frames, this is beneficial for subsequent feature processing of the first type of information in the first image.

[0085] Based on the above, this application provides an optional implementation method for obtaining the second image from the first image, which includes the following steps 401 to 404.

[0086] Step 401: Obtain the first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps.

[0087] Step 402: Invoke the first model, take the first image as the input of the first model, and output the second image; the first model includes multiple sub-models executed in sequence, each sub-model includes a first functional module for performing feature extraction, a second functional module for performing feature fusion, and a third functional module for performing nonlinear transformation.

[0088] In the embodiments of this application, the first model is an improved model based on the Mamba model, which is a neural network architecture derived from the state-space model.

[0089] For example, such as Figure 7 As shown, the first model includes multiple sub-models that are executed serially. Each sub-model includes a first functional module, a second functional module, and a third functional module. The second functional module is obtained by replacing the original SSD module with an NC-SSD, and the third functional module is an FFN module.

[0090] For example, the first model also includes a fourth functional module, such as Figure 4 The LGAG module shown performs feature enhancement processing on the outputs of multiple sub-models respectively.

[0091] Optionally, the sub-model also includes a local perceptual unit before the NC-SSD and FFN modules, which can enhance the sub-model's ability to perceive local features.

[0092] In one possible implementation, step 402 further includes steps 412 and 422, specifically regarding how to obtain the second image from the first image.

[0093] Step 412: Input the first image into the first model and output multiple first intermediate features from multiple sub-models.

[0094] In this embodiment, the first intermediate feature is obtained by performing feature extraction, feature fusion, and nonlinear transformation on the first type of information in the first image.

[0095] For example, such as Figure 7As shown, the sub-model adopts the VSSD model. The first intermediate feature output by the VSSD model at each level serves as both the input to the next level of the VSSD model and the output of that VSSD model. Therefore, multiple sub-models executed in sequence will output multiple first intermediate features. Generally, the first intermediate features of the top three levels of sub-models are used for feature enhancement processing in step 422.

[0096] Step 422: Perform feature enhancement processing on each first intermediate feature to obtain the second image.

[0097] For example, three different levels of first intermediate features x The inputs are respectively fed into the LGAG module, utilizing the gating signals of the LGAG module. g This is used to control the flow of information at different stages of the network and improve the accuracy of image segmentation.

[0098] Alternatively, compared to using 1×1 convolution to process gated signals and input features, the LGAG module uses 3×3 grouped convolution because 3×3 convolution has a large receptive field, which can capture a larger range of local spatial context, which is beneficial for feature focusing in key areas.

[0099] For example, in q att In the () function, the gating signal is controlled. g and the first intermediate feature x Apply 3×3 grouped convolution respectively GC g ()and GC x () is processed, and then normalized () BN These features are processed and merged with the generated feature map through element-wise addition. The ReLU function is then used to apply this feature map. R Activate it with ()) to obtain q att The function () can be expressed as the formula (1) shown below.

[0100] Formula (1) Furthermore, according to formula (1), a 1×1 convolution is used ( C The `followedbyBN()layer` method obtains a single-channel feature map. This generated single-channel feature map is then input into the activation function `Sigmoid(σ())` to generate attention coefficients. These attention coefficients are then scaled element-wise to optimize the first intermediate feature map. x This leads to the generation of attention-gated features LGAG ( g , x ), can be expressed as formula (2) as shown below.

[0101] Formula (2) Step 403: Based on the second image, determine the road surface type of the road where the vehicle is located; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface.

[0102] Step 404: Determine the vehicle's road adhesion coefficient based on the road surface type.

[0103] The specific steps of steps 401, 403 to 404 can be found in steps 201, 203 to 204, and will not be repeated here.

[0104] In this embodiment, the SSD is replaced with an NC-SSD to develop the VSSD module in the network architecture, improving the module's computational efficiency and accuracy. A feedforward network (FFN) is integrated after the NC-SSD module to facilitate cross-channel information exchange. Furthermore, in the LGAG module, [the following is also mentioned:] q att The () function uses 3×3 grouped convolutions, and the LGAG module is able to capture a relatively large spatial context at a low computational cost.

[0105] Building upon the above, and addressing the issue of how to obtain the first intermediate feature from the first image based on multiple sub-models executed serially in the first model, embodiments of this application provide an optional implementation method, such as... Figure 5 The flowchart shown is another method for acquiring a second image, and the specific method includes the following steps 501 to 507.

[0106] Step 501: Obtain the first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps.

[0107] Step 502: Input the first image into the first functional module of the sub-model of the first model, and output multiple features, including: the first feature obtained by feature mapping processing, and the second, third and fourth features obtained by feature mapping and nonlinear transformation.

[0108] In the embodiments of this application, the first feature is obtained by performing feature mapping processing on the first image; the second, third, or fourth feature is obtained by feature mapping and nonlinear transformation of the first image.

[0109] For example, such as Figure 7As shown, the first image is input into the sub-model. It first undergoes distribution normalization through a dual-path layer normalization (LN) layer. The resulting two outputs are then input into two linear transformation layers to complete the feature space mapping. One linear transformation layer outputs two paths: one is the first feature A, and the other undergoes local feature extraction through a convolutional layer (Conv), followed by nonlinear processing using an activation function (σ), outputting three branches: the second feature X, the third feature B, and the fourth feature C. The other linear transformation layer outputs one initial feature Z.

[0110] Step 503: Input multiple features into the second functional module of the sub-model and output the second intermediate feature. The second functional module includes the NC-SSD module, which is used to indicate that feature fusion processing is performed on multiple features.

[0111] For example, such as Figure 7 As shown, the first feature A, the second feature X, the third feature B, and the fourth feature C obtained in step 502 are all input into the NC-SSD module to perform multimodal feature fusion and adaptive weighted calculation processing, and output the second intermediate feature Y.

[0112] Optionally, after the second intermediate feature Y is processed by a Layer Normalization (LN) layer, it is added to the initial feature Z through a skip connection to perform a residual addition operation. After being processed by a Linear Transformation (Linear) layer, it is added to the first image through a skip connection to perform a residual addition operation to obtain the second intermediate feature.

[0113] Step 504: Input the second intermediate feature into the third functional module of the sub-model and output the first intermediate feature; the third functional module includes: FNN module, which is used to indicate: perform nonlinear transformation processing on the second intermediate feature.

[0114] For example, the second intermediate feature first passes through a layer normalization (LN) layer to achieve a stable gradient distribution of the feature, and then is input into the FNN module for nonlinear transformation to generate output features with spatial-semantic comprehensive representation capabilities.

[0115] Step 505: Use the first intermediate feature as input to the fourth functional module of the first model and output the second image; the fourth functional module is used to indicate: perform feature enhancement processing on the first intermediate feature.

[0116] Step 506: Based on the second image, determine the road surface type of the road where the vehicle is located; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface.

[0117] Step 507: Determine the road adhesion coefficient of the vehicle based on the road surface type.

[0118] For a detailed description of steps 501, 506 to 507, please refer to steps 201, 203 to 204. For a detailed description of step 505, please refer to step 422. These steps will not be repeated here.

[0119] In this embodiment of the application, for the sub-modules in the first module, the entire architecture achieves cross-modal feature interaction while ensuring gradient stability through residual connections, normalization layer stacking and multi-path fusion mechanisms, forming a feature learning unit with end-to-end optimization capabilities.

[0120] Based on the above, this application provides an optional implementation method for determining the vehicle's road adhesion coefficient, such as... Figure 6 The diagram shows a method for obtaining the road surface adhesion coefficient, which includes the following steps 601 to 605.

[0121] Step 601: Obtain the first image of the vehicle; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps.

[0122] Step 602: Perform feature processing on the first type of information in the first image to obtain the second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires.

[0123] Step 603: Perform image fusion processing on multiple second images to obtain the road surface type of the road where the vehicle is located.

[0124] For example, multiple first intermediate features representing different capabilities are respectively input into the fourth functional module of the first model, and correspondingly output second images at multiple scales. After feature fusion processing is performed on the second images at multiple scales, the road surface type in the feature fusion-processed second image is identified.

[0125] Step 604: Determine the first threshold of the road surface adhesion coefficient based on the vehicle's body mass and the water depth of the road surface.

[0126] In this embodiment of the application, the first threshold is determined based on the vehicle's own mass and the water depth of the road, and is a threshold related to the vehicle's driving speed.

[0127] Optionally, based on real-vehicle dynamic excitation tests, the first threshold can be obtained as follows: ,in, m For vehicle body quality, s This refers to the depth of the water surface.

[0128] Step 605: Determine the vehicle's road adhesion coefficient based on the first threshold, the vehicle's driving speed, and the road surface type.

[0129] For example, when a vehicle is on a wet or waterlogged asphalt road, it is prone to hydroplaning at certain speeds, resulting in a significant reduction in the peak value of TRFC. This reduction is affected by vehicle weight, driving speed, and water depth.

[0130] When a vehicle is on a wet asphalt road surface, the road adhesion coefficient is calculated using formula (3).

[0131] Formula (3) in, The coefficient of adhesion for wet asphalt pavement. v For the vehicle's speed, m For vehicle body quality, s This refers to the depth of the water surface.

[0132] When a vehicle is on a waterlogged asphalt road, the road adhesion coefficient is calculated using formula (4).

[0133] Formula (4) in, This is the road adhesion coefficient under waterlogged asphalt pavement.

[0134] For example, when a vehicle is on a muddy or snow-melted muddy road surface, at low speeds, the road adhesion coefficient of the snow-melted muddy road surface is comparable to that of a medium-adhesion road surface. As the vehicle speed increases, the road adhesion coefficient decreases significantly.

[0135] When a vehicle is on a muddy or snow-melted muddy road surface, the road surface adhesion coefficient is calculated using formula (5).

[0136] Formula (5) in, The coefficient of adhesion for muddy or snow-melted muddy surfaces.

[0137] The specific steps of steps 601 to 602 can be found in steps 201 to 202, and will not be repeated here.

[0138] To achieve the functions of the above embodiments, the data processing method includes hardware structures and / or software modules corresponding to each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed through hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0139] exist Figures 2 to 7 Based on the data processing method shown, the present application also provides a data processing apparatus for further description, such as... Figure 8 The schematic diagram of the data processing device shown includes: a first acquisition module 810, a first processing module 820, a second processing module 830, and a third processing module 840.

[0140] The first acquisition module 810 is used to acquire a first image of the vehicle; the first image is obtained by fusing multiple frame images of the vehicle's tires at multiple timestamps; wherein, the first acquisition module 810 may include, for example, Figure 1 The first module 101 and the second module 102 are shown.

[0141] The first processing module 820 is used to perform feature processing on a first type of information in the first image to obtain a second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires; wherein, the first processing module 820 may include, for example, Figure 1 The second module 102 shown.

[0142] The second processing module 830 is used to determine the road surface type of the road where the vehicle is located based on the second image; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface; wherein, the second processing module 830 may include, for example, Figure 1 The second module 102 shown.

[0143] The third processing module 840 is used to determine the vehicle's road adhesion coefficient based on the road surface type. The third processing module 840 may include, for example: Figure 1 The second module 102 shown.

[0144] In some embodiments, the first acquisition module 810 includes: acquiring a first initial image of a vehicle with multiple timestamps; the first initial image includes a first type of information and a second type of information; the first type of information includes road surface, lane lines, and vehicle tires; the second type of information includes buildings and vegetation; processing the first initial image to obtain a first target image, the first target image being used to indicate: determining the depth information of each pixel in the first initial image; and stitching the first target image according to the chronological order to obtain a first image.

[0145] In some embodiments, the first processing module 820 includes: calling a first model, taking a first image as input to the first model, and outputting a second image; the first model includes multiple sub-models executed sequentially, each sub-model including a first functional module for performing feature extraction, a second functional module for performing feature fusion, and a third functional module for performing nonlinear transformation.

[0146] In some embodiments, the first processing module 820 further includes: inputting a first image into a first model, outputting a plurality of first intermediate features from a plurality of sub-models; and performing feature enhancement processing on each first intermediate feature to obtain a second image.

[0147] In some embodiments, the first processing module 820 further includes: a first functional module for inputting the first image into a sub-model of the first model and outputting multiple features, the multiple features including: a first feature obtained by feature mapping processing, and a second feature, a third feature, and a fourth feature obtained by feature mapping and nonlinear transformation; a second functional module for inputting the multiple features into the sub-model and outputting a second intermediate feature, the second functional module including: an NC-SSD module, the NC-SSD module being used to instruct: performing feature fusion processing on the multiple features; a third functional module for inputting the second intermediate feature into the sub-model and outputting a first intermediate feature; the third functional module including: an FNN module, the FNN module being used to instruct: performing nonlinear transformation processing on the second intermediate feature.

[0148] In some embodiments, the first processing module 820 further includes: using the first intermediate feature as input to a fourth functional module of the first model, and outputting a second image; the fourth functional module is used to instruct: to perform feature enhancement processing on the first intermediate feature.

[0149] In other embodiments, the second processing module 830 includes: performing image fusion processing on multiple second images to obtain the road surface type of the road where the vehicle is located.

[0150] In some embodiments, the third processing module 840 includes: determining a first threshold for the road surface adhesion coefficient based on the vehicle's body mass and the water depth of the road surface; and determining the vehicle's road surface adhesion coefficient based on the first threshold, the vehicle's driving speed, and the road surface type.

[0151] According to one aspect of the embodiments of this application, Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 includes a processor 910 and one or more memories 920. The one or more memories 920 are used to store program instructions executed by the processor 910. When the processor 910 executes the program instructions, it implements the above-described interface processing method.

[0152] Furthermore, the processor 910 may include one or more processing cores. The processor 910 runs or executes instructions, programs, code sets, or instruction sets stored in the memory 920, and retrieves data stored in the memory 920. Optionally, the processor 910 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 910 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor and may be implemented using a separate communication chip.

[0153] According to one aspect of this application, a computer-readable storage medium is also provided, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0154] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0155] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0158] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: A first image of the vehicle is acquired; the first image is obtained by fusing multiple frames of images of the vehicle's tires at multiple timestamps. The first type of information in the first image is processed to obtain the second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires; Based on the second image, the road surface type of the road where the vehicle is located is determined; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface. The road adhesion coefficient of the vehicle is determined based on the road surface type.

2. The method according to claim 1, characterized in that, The acquisition of the first image of the vehicle includes: Acquire a first initial image of the vehicle with multiple timestamps; the first initial image includes a first type of information and a second type of information; the first type of information includes road surface, lane lines, and vehicle tires; the second type of information includes buildings and vegetation; The first initial image is processed to obtain a first target image, which is used to indicate: the depth information of each pixel in the first initial image is determined; The first target image is obtained by stitching the images together in chronological order.

3. The method according to claim 1, characterized in that, The step of performing feature processing on the first image to obtain the second image includes: The first model is invoked, the first image is used as the input of the first model, and the second image is output; the first model includes multiple sub-models executed in sequence, each sub-model including a first functional module for performing feature extraction, a second functional module for performing feature fusion and a third functional module for performing nonlinear transformation.

4. The method according to claim 3, characterized in that, The first model is invoked, with the first image as input, and the second image is output. The first model includes multiple sub-models executed sequentially, including: The first image is input into the first model, and multiple first intermediate features are output from the multiple sub-models; The second image is obtained by performing feature enhancement processing on each of the first intermediate features.

5. The method according to claim 4, characterized in that, The step of inputting the first image into the first model and outputting multiple first intermediate features from multiple sub-models includes: The first image is input into the first functional module of the sub-model of the first model, and multiple features are output. The multiple features include: a first feature obtained by feature mapping processing, and a second, third and fourth feature obtained by feature mapping and nonlinear transformation. The multiple features are input into the second functional module of the sub-model, and the second intermediate feature is output. The second functional module includes an NC-SSD module, which is used to instruct: to perform feature fusion processing on the multiple features. The second intermediate feature is input into the third functional module of the sub-model, and the first intermediate feature is output. The third functional module includes an FNN module, which is used to indicate that the second intermediate feature is subjected to nonlinear transformation processing.

6. The method according to claim 4, characterized in that, The second image is obtained by performing feature enhancement processing on each of the first intermediate features, including: The first intermediate feature is used as input to the fourth functional module of the first model to output a second image; the fourth functional module is used to instruct: to perform feature enhancement processing on the first intermediate feature.

7. The method according to claim 1, characterized in that, Determining the road surface type of the road where the vehicle is located based on the second image includes: Image fusion processing is performed on multiple second images to obtain the road surface type of the road where the vehicle is located.

8. The method according to claim 7, characterized in that, Determining the road adhesion coefficient of the vehicle based on the road surface type includes: A first threshold for the road surface adhesion coefficient is determined based on the vehicle's body mass and the water depth of the road surface. The road adhesion coefficient of the vehicle is determined based on the first threshold, the vehicle's driving speed, and the road surface type.

9. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a first image of the vehicle; the first image is obtained by fusing multiple frame images of the vehicle's tires at multiple timestamps; A first processing module is used to perform feature processing on a first type of information in the first image to obtain a second image; the feature processing includes feature fusion and feature enhancement, and the first type of information includes: road surface, lane lines, and vehicle tires; The second processing module is used to determine the road surface type of the road where the vehicle is located based on the second image; the road surface type includes at least one of the following: dry asphalt road surface, semi-wet asphalt road surface, waterlogged asphalt road surface, snow-covered road surface, rough icy road surface, or smooth icy road surface. The third processing module is used to determine the road adhesion coefficient of the vehicle based on the road surface type.

10. A computer device, characterized in that, The computer device includes: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the data processing method as described in any one of claims 1 to 8.

Images