Vehicle control method and apparatus based on multi-modal data and vehicle

Abstract

The embodiment of the application provides a kind of vehicle control method, device and vehicle based on multi-modal data, it is related to intelligent driving technical field.The method comprises: obtaining the multi-modal perception data of vehicle, multi-modal perception data contains forward-looking video stream, vehicle state data and tire road surface action data;Based on forward-looking video stream, vehicle state data and tire road surface action data, multi-modal fusion multi-task processing is carried out, and the drivable area, road type and tire wear state corresponding to the multi-modal perception data of vehicle are obtained;According to drivable area and road type, determine terrain risk value;According to tire wear state and terrain risk value, determine tire terrain coupling risk value;Based on tire terrain coupling risk value, whole vehicle dynamic control is carried out to vehicle.The method is used to reach the effect of improving the reliability of all-terrain intelligent driving control.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a vehicle control method, device and vehicle based on multimodal data. Background Technology

[0002] Intelligent driving is a technology that uses environmental perception, intelligent decision-making, and control systems to take over some or all of the driving tasks in specific scenarios, thereby improving driving safety and travel efficiency.

[0003] In related technologies, intelligent driving relies on clear lane lines and traffic signs for intelligent decision-making and performs well on structured roads. However, in real-world driving scenarios, vehicles may have to deal with various road surfaces such as mud, gravel, ice, snow, water, and sand. In some terrains, intelligent driving suffers from poor reliability and is unable to meet the needs of all-terrain driving. Summary of the Invention

[0004] This application provides a vehicle control method, device, and vehicle based on multimodal data to improve the reliability of all-terrain intelligent driving control.

[0005] In a first aspect, embodiments of this application provide a vehicle control method based on multimodal data, including:

[0006] Acquire multimodal perception data of the vehicle, which includes forward-looking video stream, vehicle status data, and tire-road interaction data;

[0007] Based on forward-looking video stream, vehicle status data, and tire-road interaction data, multi-modal fusion and multi-task processing are performed to obtain the drivable area, road type, and tire wear status of the vehicle's corresponding multi-modal perception data.

[0008] Determine the terrain risk value based on the drivable area and road surface type;

[0009] The tire-terrain coupling risk value is determined based on the tire wear condition and terrain risk value;

[0010] Based on the tire-terrain coupling risk value, the vehicle is dynamically controlled.

[0011] Secondly, embodiments of this application provide a vehicle cooperative control system, including:

[0012] Control components and multiple subsystems;

[0013] The control component is used to execute vehicle control methods based on multimodal data; the control component interacts with multiple subsystems of the vehicle to achieve dynamic control of the entire vehicle.

[0014] Thirdly, embodiments of this application provide a vehicle control device based on multimodal data, comprising:

[0015] The acquisition unit is used to acquire multimodal perception data of the vehicle, which includes forward-looking video stream, vehicle status data, and tire-road interaction data.

[0016] The multi-task processing unit is used to perform multi-modal fusion processing based on forward-looking video stream, vehicle status data and tire road surface action data to obtain the drivable area, road surface type and tire wear status of the vehicle corresponding to the multi-modal perception data.

[0017] The terrain risk determination unit is used to determine the terrain risk value based on the drivable area and road surface type.

[0018] The coupling risk determination unit is used to determine the tire-terrain coupling risk value based on the tire wear condition and terrain risk value;

[0019] The dynamic control unit is used to perform overall vehicle dynamic control based on tire-terrain coupling risk values.

[0020] Fourthly, embodiments of this application provide a vehicle, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0022] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0023] The vehicle control method, device, and vehicle based on multimodal data provided in this application integrate forward-looking video streams, vehicle state data, and tire-road interaction data through multimodal fusion multi-task processing to obtain drivable area, road surface type, and tire wear status. This achieves dynamic correlation between drivable area, road surface type, and tire wear status, avoiding the separation of tire wear status from other sensing data and improving the accuracy of drivable area, road surface type, and tire wear status. Furthermore, the terrain risk value is determined based on the drivable area and road surface type, allowing the terrain risk value to reflect the risk of vehicle passage through the current terrain, thus improving the reliability of the terrain risk value. Reliability is paramount, as tires are the only point of contact between a vehicle and the road surface. Tire condition significantly impacts driving safety and passability. By determining the tire-terrain coupling risk value based on terrain risk and tire wear condition, this value reflects the dynamic interactive risks of tires operating in the current terrain environment, including the current road surface type and drivable area, thus improving the accuracy of tire-terrain coupling risk assessment. Vehicle dynamic control based on the tire-terrain coupling risk value breaks down the data silos between tire detection and vehicle control, enhancing driving safety and passability in complex terrains and improving the reliability of all-terrain intelligent driving. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 A flowchart illustrating the vehicle control method based on multimodal data provided in this application;

[0026] Figure 2 A schematic diagram illustrating the setup of the tire sensor and the triaxial acceleration sensor provided in this application;

[0027] Figure 3 A schematic diagram of the drivable area provided in this application;

[0028] Figure 4 The flowchart provided in this application illustrates the process of determining drivable areas, road surface types, and tire wear conditions using a multi-task model based on multimodal fusion. Figure 1 ;

[0029] Figure 5 Visual renderings of the road surface types provided in this application;

[0030] Figure 6 A schematic diagram illustrating the extraction of temporal synchronization features and enhancement of video features provided in this application;

[0031] Figure 7 A schematic diagram illustrating the extraction and enhancement of video features provided in this application;

[0032] Figure 8 A schematic diagram illustrating the extraction of local enhancement features provided in this application;

[0033] Figure 9 The flowchart provided in this application illustrates the process of determining drivable areas, road surface types, and tire wear conditions using a multi-task model based on multimodal fusion. Figure 2 ;

[0034] Figure 10 A schematic diagram of the vehicle cooperative system provided in this application;

[0035] Figure 11 A schematic diagram of the control components provided in this application;

[0036] Figure 12 A schematic diagram of the structure of the vehicle control device based on multimodal data provided in this application;

[0037] Figure 13 A structural schematic diagram of the vehicle provided in this application.

[0038] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] Intelligent driving is a technology that uses environmental perception, intelligent decision-making, and control systems to take over some or all of the driving tasks in specific scenarios, thereby improving driving safety and travel efficiency.

[0041] In related technologies, intelligent driving relies on clear lane lines and traffic signs for intelligent decision-making and performs well on structured roads. However, in real-world driving scenarios, vehicles may have to deal with various road surfaces such as mud, gravel, ice, snow, water, and sand. In some terrains, intelligent driving suffers from poor reliability. For example, driving on soft sand can reduce passability, while driving on hard surfaces can lead to severe tire wear, making it difficult to meet the needs of all-terrain driving.

[0042] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart illustrating the vehicle control method based on multimodal data provided in this application. The vehicle control method based on multimodal data can be applied to vehicles, such as to the vehicle's on-board controller. Figure 1 As shown, the vehicle control method based on multimodal data includes:

[0044] S101. Acquire multimodal perception data of the vehicle, which includes forward-looking video stream, vehicle status data, and tire-road interaction data.

[0045] Among them, the forward-view video stream is captured by the vehicle's forward-view camera, whose field of view covers the road surface in front of the vehicle.

[0046] Vehicle status data reflects the vehicle's motion state. Vehicle status data includes three-axis acceleration, tire status data, and vehicle speed. Three-axis acceleration includes longitudinal acceleration, lateral acceleration, and vertical acceleration. Tire status data includes tire pressure and tire temperature.

[0047] In practical applications, triaxial acceleration is detected by the vehicle's triaxial acceleration sensor; tire pressure and tire temperature are detected by the vehicle's tire pressure monitoring system (TPMS).

[0048] Among them, tire-road interaction data is used to reflect the changes in tire state caused by the interaction between the tire and the road surface. The tire-road interaction data is collected by tire sensors installed on the tire. The tire sensors can be piezoelectric sensors. The tire-road interaction data includes electrical signals. By collecting electrical signals over a period of time, the pressure changes, strain changes and vibration conditions at the corresponding positions on the tire (where the piezoelectric sensors are located) can be determined.

[0049] In practical applications, tire sensors can be PVDF (Polyvinylidene Fluoride) piezoelectric sensors. Multiple PVDF piezoelectric sensors are set on the inner surface of the tire. The PVDF piezoelectric sensors output electrical signals. For a certain PVDF piezoelectric sensor, the electrical signal of the PVDF piezoelectric sensor over a period of time can reflect the pressure changes, strain changes and vibration conditions at the location of the PVDF piezoelectric sensor.

[0050] In a specific example, such as Figure 2 As shown, a data acquisition microcontroller unit (MCU), a triaxial accelerometer, and five embedded PVDF piezoelectric sensors are installed on the inner side of the tire. The embedded PVDF piezoelectric sensors collect tire road surface action data and send it to the MCU. The triaxial accelerometers collect triaxial acceleration data and send it to the MCU. The MCU then sends the tire road surface action data and triaxial acceleration data to the vehicle-side controller.

[0051] Specifically, it continuously acquires forward-view video streams from the forward-view camera, tire-road interaction data from the tire sensor, triaxial acceleration from the triaxial accelerometer, and tire pressure and temperature from the tire pressure monitoring system.

[0052] S102. Based on the forward-looking video stream, vehicle status data, and tire road surface action data, perform multi-modal fusion multi-task processing to obtain the drivable area, road surface type, and tire wear status of the vehicle's corresponding multi-modal perception data.

[0053] The drivable area refers to the area within the field of view of the forward-looking camera that the vehicle can pass through; the forward-looking video stream includes multiple forward-looking video frames, each of which corresponds to a drivable area; in practical applications, the drivable area is a portion of the segmented mask image corresponding to the forward-looking video frame.

[0054] Road surface type is used to reflect the classification of terrain; tire wear condition refers to the degree of wear on the tire surface.

[0055] For example, road surface types include, but are not limited to: muddy roads, gravel roads, snow, ice, dry asphalt roads, and wet asphalt roads; tire wear condition can be represented by tire wear value. The larger the tire wear value, the heavier the wear on the tire surface, and the smaller the tire wear value, the lighter the wear on the tire surface.

[0056] In one possible implementation, for the forward-looking video stream, vehicle status data, and tire-road interaction data acquired within the same time period, obstacle and drivable area boundaries are identified based on the forward-looking video stream, and the drivable area is obtained through pixel-level segmentation processing.

[0057] Multimodal fusion is used to extract features from the forward-looking video stream, vehicle status data, and tire-road interaction data. Road surface detection is performed based on the multimodal fusion features to obtain the road surface type, and tire wear detection is performed based on the multimodal fusion features to obtain the tire wear status.

[0058] In one possible implementation, multimodal fusion feature extraction is performed on the forward-looking video stream, vehicle state data, and tire-road interaction data to obtain multimodal fusion features. Image segmentation is then performed based on the multimodal fusion features to obtain the drivable area. Road surface detection is then performed based on the multimodal fusion features to obtain the road surface type. Finally, tire wear detection is performed based on the multimodal fusion features to obtain the tire wear status.

[0059] In practical applications, a multi-task detection model is used for multimodal fusion processing. The multi-task detection model includes a feature extraction module, an image segmentation module, a road surface detection module, and a tire wear detection module. The feature extraction module extracts multimodal fusion features corresponding to the forward-looking video stream, vehicle state data, and tire-road interaction data. The image segmentation module processes the multimodal fusion features to obtain the drivable area. The road surface detection module processes the multimodal fusion features to obtain the road surface type. The tire wear detection module processes the multimodal fusion features to obtain the tire wear status.

[0060] For example, such as Figure 3 As shown, Figure 3 (a) is a video frame in the forward-looking video stream. Figure 3 In (b), the masked portion (i.e., the blue portion) is the drivable area corresponding to that video frame.

[0061] S103. Determine the terrain risk value based on the drivable area and road surface type.

[0062] Among them, the terrain risk value can reflect the risk of a vehicle traveling in the terrain formed by the current drivable area and road surface type.

[0063] In one possible implementation, a basic risk value corresponding to the road surface type is determined, a traffic adjustment value is determined based on the area of ​​the drivable area, and the basic risk value is adjusted using the traffic adjustment value to obtain the terrain risk value; for example, the product of the traffic adjustment value and the basic risk value is used as the terrain risk value.

[0064] Specifically, if the area of ​​the drivable zone belongs to the first area range, the traffic adjustment value is determined to be the first coefficient; if the area of ​​the drivable zone belongs to the second area range, the traffic adjustment value is determined to be the second coefficient; and if the area of ​​the drivable zone belongs to the third area range, the traffic adjustment value is determined to be the third coefficient.

[0065] Among them, the first area interval, the second area interval, the third area interval, the first coefficient, the second coefficient, and the third coefficient can all be set according to actual needs; the area of ​​the first area interval is greater than the area of ​​the second area interval, the first coefficient is less than the second coefficient, and the first coefficient is less than 1; the area of ​​the second area interval is greater than the area of ​​the third area interval, the second coefficient is less than the third coefficient, and the third coefficient is greater than 1.

[0066] In other words, if the drivable area is large, the basic risk value is reduced by adjusting the traffic flow; if the drivable area is small, the basic risk value is increased by adjusting the traffic flow.

[0067] In one possible implementation, a basic risk value corresponding to the road surface type is determined, a traffic adjustment value is determined based on the area of ​​the drivable area, and a dynamic adjustment value is determined based on the vehicle's dynamic data. The basic risk value is then adjusted using the traffic adjustment value and the dynamic adjustment value to obtain the terrain risk value. For example, the product of the basic risk value, the traffic adjustment value, and the dynamic adjustment value can be used as the terrain risk value.

[0068] The vehicle's dynamic data includes, but is not limited to: vehicle speed, three-axis acceleration, steering angle, drive torque, braking status, gradient, ambient temperature, and system running time. In practical applications, three-axis acceleration can be detected by a three-axis acceleration sensor, and gradient can be detected by an inertial measurement unit (IMU). The above dynamic data can be acquired in real time through the vehicle's CAN bus.

[0069] Specifically, the vehicle's motion state is determined based on the vehicle's dynamic data. If the motion state is aggressive, the dynamic adjustment value is the fourth coefficient; if the motion state is conservative, the dynamic adjustment value is the fifth coefficient. The fourth coefficient is greater than 1, and the fifth coefficient is less than 1.

[0070] The motion state of the vehicle is determined based on at least one of the following: vehicle speed, three-axis acceleration, steering angle, braking status, and slope.

[0071] For example, if the vehicle speed is greater than the preset speed and / or the steering angle is greater than the preset steering angle, the motion state is determined to be aggressive; if the vehicle speed is not greater than the preset speed and the steering angle is greater than the preset steering angle, the motion state is determined to be conservative; the preset speed and preset steering angle are set according to actual needs.

[0072] For example, if any of the three-axis accelerations is greater than the corresponding acceleration threshold, and / or the steering angle is greater than the preset steering angle, the motion state is determined to be aggressive; otherwise, the motion state is determined to be conservative. The acceleration threshold is set according to actual needs.

[0073] S104. Determine the tire-terrain coupling risk value based on the tire wear condition and terrain risk value.

[0074] The tire-terrain coupling risk value reflects the dynamic interaction risk of a tire operating in the current wear condition within the terrain environment consisting of the current road surface type and drivable area. A higher tire-terrain coupling risk value indicates a higher dynamic interaction risk for the tire operating in the current wear condition within the terrain environment consisting of the current road surface type and drivable area. Conversely, a lower tire-terrain coupling risk value indicates a lower dynamic interaction risk for the tire operating in the current wear condition within the terrain environment consisting of the current road surface type and drivable area.

[0075] Specifically, the tire health value is determined based on the tire wear condition, and the tire terrain coupling risk value is determined based on the tire health value and the terrain risk value.

[0076] Optionally, the tire wear condition is represented by the tire wear value. The tire health value is determined based on the tire wear condition value, and the tire risk coefficient is determined based on the tire health value. The tire health value and the tire risk coefficient are negatively correlated. The tire terrain coupling risk value is determined based on the terrain risk value and the tire risk coefficient. For example, the product between the terrain risk value and the tire risk coefficient is used as the tire terrain coupling risk value.

[0077] Among them, tire wear value and tire health value are negatively correlated. For example, the tire wear value is a value between 0 and 1, and the tire health value is a value between 1 and 100. If the tire wear value is 0.2, the tire health value is 80, which means that the tire wear is not serious and the tire is in good health.

[0078] Optionally, the tire aging value is determined based on the tire's usage time, and the tire wear value and tire aging value are weighted and summed. Then, the tire health value is determined based on the weighted summation result.

[0079] S105. Based on the tire-terrain coupling risk value, perform dynamic control of the vehicle as a whole.

[0080] This includes dynamic control of the vehicle as a whole, which includes controlling any subsystem of the vehicle; for example, controlling the power system, the energy recovery system, and the chassis system based on the tire-terrain coupling risk value.

[0081] Specifically, based on preset fuzzy rules, the terrain-tired terrain coupling risk value is mapped to subsystem control parameters, and the subsystem is controlled through these subsystem control parameters.

[0082] For example, the risk conditions corresponding to the terrain coupling risk value of the terrain tire are determined according to preset fuzzy rules; in low-risk conditions, the chassis system is controlled to work according to the suspension parameters and tire pressure corresponding to the low-risk conditions, for example, the suspension damping coefficient is set to the first damping coefficient and the tire pressure is set to the standard tire pressure; the energy recovery system is controlled to work according to the recovery mode corresponding to the low-risk conditions, for example, the recovery torque is set to the first recovery torque.

[0083] In medium-risk operating conditions, the driving torque demand of the power system is adjusted. For example, the driving torque demand is limited according to the first ratio, and the chassis system is controlled to work according to the suspension parameters and tire pressure corresponding to the medium-risk operating conditions. For example, the suspension damping coefficient is set to the second damping coefficient, and the standard tire pressure is finely adjusted according to the road surface type. The energy recovery system is controlled to work according to the recovery mode corresponding to the medium-risk operating conditions. For example, the recovery torque is set to the second recovery torque.

[0084] The second damping coefficient is greater than the first damping coefficient, and the second recovery torque is less than the first recovery torque.

[0085] In high-risk operating conditions, the driving torque demand of the power system is adjusted, for example, the driving torque demand is limited according to the second ratio, and the chassis system is controlled to work according to the suspension parameters and tire pressure corresponding to the high-risk operating conditions. For example, the suspension damping parameter is set to the third damping coefficient, and the tire pressure is actively adjusted according to the road surface type. The energy recovery system is controlled to work according to the recovery mode corresponding to the high-risk operating conditions. For example, the recovery torque is set to the third recovery torque.

[0086] Among them, the third damping coefficient is greater than the second damping coefficient, and the third recovery torque is less than the second recovery torque.

[0087] In one possible implementation, vehicle dynamic control also includes controlling the thermal management system. Specifically, the target torque demand is determined based on the tire terrain coupling risk value and the driving torque demand, and thermal management of the battery and motor is performed based on the target torque demand and the ambient temperature. For example, if the target torque demand is large and the ambient temperature is high, the starter battery is actively cooled and the coolant flow rate of the motor is increased; if the target torque demand is small and the ambient temperature is low, the battery cooling power is reduced and the motor cooling mode is set to an economical cooling mode.

[0088] In related technologies, TPMS can detect tire pressure and tire temperature, but it cannot accurately assess the tire's operating status and wear. Furthermore, tire pressure and tire temperature are only used as isolated data to remind users to pay attention to the tire's condition. The tire's health status is not integrated with other sensory data for in-depth analysis, nor is it incorporated into vehicle control decisions. This results in the tire's health status being disconnected from other sensory data and system decisions, making it difficult for the vehicle to meet all-terrain driving requirements.

[0089] The vehicle control method based on multimodal data provided in this application integrates forward-looking video streams, vehicle state data, and tire-road interaction data through multimodal fusion and multi-task processing to obtain drivable area, road surface type, and tire wear status. This achieves dynamic correlation between drivable area, road surface type, and tire wear status, avoiding the separation of tire wear status from other sensing data and improving the accuracy of drivable area, road surface type, and tire wear status. Furthermore, the method determines terrain risk values ​​based on the drivable area and road surface type, ensuring that the terrain risk value reflects the risk of vehicle passage through the current terrain and improving the reliability of the terrain risk value. Tires are the only point of contact between a vehicle and the road surface, and their condition has a significant impact on driving safety and passability. By determining the tire-terrain coupling risk value based on terrain risk and tire wear condition, this value can reflect the dynamic interactive risks of tires operating in the current terrain environment, including the current road surface type and drivable area, thus improving the accuracy of tire-terrain coupling risk assessment. Based on this risk value, dynamic vehicle control is implemented, breaking down the data silos between tire detection and vehicle control, improving driving safety and passability in complex terrains, and enhancing the reliability of all-terrain intelligent driving.

[0090] In some embodiments, multi-modal fusion multi-task processing is performed based on forward-looking video stream, vehicle state data, and tire-road interaction data to obtain the drivable area, road surface type, and tire wear state of the vehicle's corresponding multi-modal perception data. This includes: performing multi-attention feature extraction and spatiotemporal feature extraction on the forward-looking video stream to obtain enhanced video features and temporal synchronization features; performing feature extraction based on vehicle state data and tire-road interaction data to obtain signal features; performing feature fusion on the enhanced video features and signal features to obtain multi-modal fusion features; performing image segmentation processing based on the enhanced video features and temporal synchronization features to obtain the drivable area; and performing road surface detection and tire wear detection based on the multi-modal fusion features to obtain the road surface type and tire wear state.

[0091] Among them, enhanced video features are obtained by extracting spatial features and enhancing attention in the forward-looking video stream, while temporal synchronization features are obtained by extracting spatial features and performing temporal synchronization in the forward-looking video stream.

[0092] Signal features are obtained by extracting features from fused data consisting of vehicle state data and tire-road interaction data.

[0093] In one possible implementation, multimodal fusion multi-task processing is performed through a multimodal fusion multi-task model (Visual Signal-MultiTask Fusion, VS-MTF). The VS-MT model adopts a shared visual backbone network and a dual-stream processing structure. The shared visual backbone network can be implemented by an improved EfficientNet-B3 (a high-efficiency convolutional neural network). The dual streams refer to the video stream and the signal stream consisting of vehicle state data and tire road surface action data.

[0094] Specifically, refer to Figure 4 The multi-modal fusion multi-task model includes: video stream feature extraction module, signal stream feature extraction module, multimodal feature fusion module, video feature fusion module, road surface feature extraction module, tire feature extraction module, drivable area segmentation module, road surface detection module, and tire detection module.

[0095] The forward-looking video stream is input to the video stream feature extraction module, which performs multi-attention feature extraction and spatiotemporal feature extraction to obtain enhanced video features and temporal synchronization features. Based on vehicle status data and tire-road interaction data, the fused signal data is determined and input to the signal stream feature extraction module to obtain signal features.

[0096] The enhanced video features and signal features are input into the multimodal feature fusion module to obtain multimodal fusion features; the enhanced video features and temporal synchronization features are input into the video feature fusion module to obtain target video features; the multimodal fusion features are input into the road surface feature extraction module to obtain target road surface features; and the multimodal fusion features are input into the tire feature extraction module to obtain target tire features.

[0097] The target video features are input into the drivable area segmentation module to obtain the drivable area; the target road surface features are input into the road surface detection module to obtain the road surface type; and the target tire features are input into the tire detection module to obtain the tire wear status.

[0098] For example, such as Figure 5 As shown, Figure 5 (a) is a visual rendering of a snowy road surface. Figure 5 (b) is a visual rendering of a dry asphalt pavement. Figure 5 (c) is a visual rendering of a wet asphalt pavement.

[0099] The vehicle status data includes: longitudinal acceleration, lateral acceleration, vertical acceleration, vehicle speed, tire pressure, and tire temperature; the tire-road interaction data includes electrical signals.

[0100] Based on vehicle status data and tire-road interaction data, the fused signal data is determined, including: analyzing the electrical signals acquired within a time period to obtain time-domain and frequency-domain characteristics. Time-domain characteristics include: mean, variance, and number of peaks; frequency-domain characteristics include: spectral power and spectral bandwidth. Based on longitudinal acceleration, lateral acceleration, vertical acceleration, vehicle speed, tire pressure, tire temperature, electrical signals, and time-domain and frequency-domain characteristics acquired within a time period, the fused signal data is determined.

[0101] For example, the fused signal data is represented as (longitudinal acceleration, lateral acceleration, vertical acceleration, vehicle speed, tire pressure, tire temperature, electrical signal, time domain features, and frequency domain features).

[0102] The signal flow feature extraction module includes three cascaded signal feature extraction sub-modules: a fully connected layer, a batch normalization layer, an activation function layer, and an output layer.

[0103] In some embodiments, the multimodal fusion multi-task model is obtained by training an initial model using a multi-task joint training strategy that combines dynamic task weighting and progressive learning.

[0104] Specifically, dynamic task weighting refers to dynamically adjusting the loss weights based on the prediction uncertainty of each task through an uncertainty weighting strategy. For more difficult tasks (such as tire wear detection), higher weights are automatically assigned to ensure balanced learning across tasks. Experiments show that dynamic task weighting improves the convergence speed of task learning by 35% and the final performance of the model by 8.2%.

[0105] Progressive learning refers to optimizing model parameters primarily for drivable area segmentation tasks in the initial training phase, and then gradually adding model parameters for road surface type detection tasks, followed by tire wear detection tasks, to improve the stability of the model learning process.

[0106] Optionally, to address the domain shift problem caused by differences in vehicle models and sensors, an adversarial domain adaptation mechanism is introduced. This mechanism uses a domain discriminator and a gradient inversion layer to enable the model to learn domain-invariant features, thereby improving the model's generalization ability across different vehicle models.

[0107] Optionally, to meet the computing power limitations of the vehicle platform, knowledge distillation technology is used to transfer the knowledge of the large teacher model to the lightweight student model; the number of parameters of the teacher model is greater than the number of parameters of the student model; the teacher model is trained in an offline environment, and the student model learns by imitating the intermediate features and output distribution of the teacher model, thereby reducing the performance loss of the lightweight student model.

[0108] In the above embodiments, multi-attention feature extraction and spatiotemporal feature extraction are performed on the forward-looking video stream to obtain enhanced video features and temporal synchronization features. Through spatiotemporal feature extraction, the problem of temporal asynchrony between the forward-looking video stream and vehicle state data and tire-road interaction data is solved, reducing the error of cross-modal feature processing, providing high-quality input for subsequent multi-task processing, and improving the accuracy of drivable areas. Feature fusion is performed on enhanced video features and signal features, so that the multi-modal fusion features have rich fusion information of vision and signal, providing high-quality input for subsequent task processing, and improving the accuracy of road surface type and tire wear status. The three tasks of image segmentation, road surface detection, and tire wear detection are decoupled, providing the most suitable feature input for each task, realizing efficient multi-task collaboration.

[0109] In some embodiments, multi-attention feature extraction and spatiotemporal feature extraction are performed on the forward-looking video stream to obtain enhanced video features and temporal synchronization features, including: extracting features from the forward-looking video stream to obtain initial video features; fusing dual-channel features of channel attention and spatial attention on the initial video features to obtain enhanced video features; and extracting spatiotemporal features from the initial video features to obtain temporal synchronization features.

[0110] Specifically, refer to Figure 6 The video stream feature extraction module includes: a visual feature extraction submodule, an attention feature extraction submodule, and a spatiotemporal feature extraction submodule. The forward-looking video stream is input into the visual feature extraction submodule to obtain initial video features. The initial video features are then input into the attention feature extraction submodule, where dual-channel features of channel attention and spatial attention are fused to obtain enhanced video features. Finally, the initial video features are input into the spatiotemporal feature extraction submodule to obtain temporal synchronization features.

[0111] In practical applications, the visual feature extraction submodule includes cascaded convolutional layers and a visual feature extraction network. The visual feature extraction network includes five cascaded visual feature extraction units, each consisting of a cascaded convolutional layer, a batch normalization layer, an activation layer, and a cross-stage partial 2-fusion (C2F) network. The convolutional layer (Conv), batch normalization layer (BN), and activation layer (SiLU) can be referred to as the CBS network.

[0112] In practical applications, the spatiotemporal feature extraction submodule can achieve spatiotemporal alignment through the Dynamic Time Warping (DTW) algorithm and the Kalman filter algorithm.

[0113] In the above embodiments, enhanced video features are obtained by fusing dual-channel features of channel attention and spatial attention. While maintaining low computational complexity, the ability to extract key detail features is significantly improved. By extracting spatiotemporal features, the problem of time synchronization between the forward-looking video stream and vehicle state data and tire road surface interaction data is solved, reducing the error of cross-modal feature processing, providing high-quality input for subsequent multi-task processing, and improving the accuracy of drivable areas.

[0114] In some embodiments, dual-channel feature fusion of channel attention and spatial attention is performed on the initial video features to obtain enhanced video features, including: performing enhanced downsampling processing on the initial video features to obtain local enhanced features; performing convolutional attention processing on the initial video features to obtain global enhanced features; and fusing the local enhanced features and global enhanced features to obtain enhanced video features.

[0115] Specifically, the attention feature extraction submodule can be implemented through the Enhanced Downsampling-based Convolutional Attention Fusion Module (ED-CAFM), which performs dual-channel feature fusion of channel attention and spatial attention.

[0116] refer to Figure 7 ED-CAFM includes local branches, global branches, and fusion units. Local branches include: 1 1 convolutional layer, 3 3 convolutional layers and enhanced downsampling (ED) units; global branches include: 1 One convolutional layer and lightweight self-attention units; the lightweight self-attention units consist of three parallel 3 The 3 convolutional layers also include: a first fusion layer (such as a concat layer, i.e., a splicing layer), a second fusion layer (such as a concat layer), and 1 1. Convolutional layer and 3. Third fusion layer (such as concat layer).

[0117] Specifically, for global branches, through 1 The first convolutional layer performs convolution processing on the initial video features to obtain the first video features. The first video features are then processed by a lightweight self-attention unit to obtain the global enhanced features.

[0118] Among them, through the first path 3 in the lightweight self-attention unit The first video features are processed by a third convolutional layer to obtain attention Q features, which are then processed by a second convolutional layer. The first video feature is processed by three convolutional layers to obtain the attention K feature, which is then passed through a third path. The three convolutional layers process the first video features to obtain the attention V feature; the first fusion layer fuses the attention Q feature and the attention K feature to obtain the first fused feature; the second fusion layer fuses the first fused feature and the attention V feature to obtain the second fused feature; the lightweight self-attention unit includes 1 The first convolutional layer processes the second fused feature to obtain the second video feature; the third fusion layer fuses the second video feature and the first video feature to obtain the global enhanced feature.

[0119] Specifically, for local branches, the initial video features are processed through the 1st branch included in the local branch. 1 convolutional layer, 3 The convolutional processing of the three convolutional layers yields the third video feature; the third video feature is then processed by the ED unit to obtain the local enhancement feature.

[0120] Among them, reference Figure 8 The ED unit includes a pixel recombination downsampling layer, a first depthwise separable convolutional layer, a second depthwise separable convolutional layer, a third depthwise separable convolutional layer, a fourth depthwise separable convolutional layer, a stitching layer, and a convolutional layer.

[0121] The third video feature is downsampled by a pixel unshuffle layer to obtain the first, second, third and fourth channel decomposed features. This downsampling layer reduces the spatial size of the third video feature and rearranges the pixels in the reduced area to the channel dimension to increase the number of channels without losing feature information, thus achieving lossless dimension transformation.

[0122] The first channel decomposition feature is processed by a first depthwise separable convolutional layer to obtain the first depth feature; the second channel decomposition feature is processed by a second depthwise separable convolutional layer to obtain the second depth feature; the third channel decomposition feature is processed by a third depthwise separable convolutional layer to obtain the third depth feature; the fourth channel decomposition feature is processed by a fourth depthwise separable convolutional layer to obtain the fourth depth feature; the first, second, third, and fourth depth features are concatenated by a concatenation layer to obtain the depth fusion feature; and the depth fusion feature is convolved by a convolutional layer to obtain the local enhancement feature.

[0123] The fusion unit includes: a first fully connected layer, a first normalization layer, a second fully connected layer, a second normalization layer, a third fully connected layer, and 1. 1 convolutional layer.

[0124] The global augmented features are aligned and normalized through the first fully connected layer and the first normalization layer to obtain global normalized features. The local augmented features are aligned and normalized through the second fully connected layer and the second normalization layer to obtain local normalized features. The global normalized features and the local normalized features are aligned in spatial dimensions and have similar distribution characteristics.

[0125] Globally normalized features and locally normalized features are input into the third fully connected layer. This layer determines the first fusion weight for the global normalized features and the second fusion weight for the locally normalized features. The global and locally normalized features are then fused according to these weights to obtain the first fused feature. The first fused feature is then fused with the initial video features through a residual connection to obtain the second fused feature. This process enhances key features while preserving the original features. Finally, a third fully connected layer is used... The first convolutional layer performs feature compression and nonlinear transformation on the second fused feature to obtain enhanced video features.

[0126] Specifically, the third fully connected layer can adaptively determine the first fusion weight of the global normalized features and the second fusion weight of the local normalized features. Based on the complexity and feature distribution of the current scene, the fusion weights of the local and global branches can be automatically adjusted. For example, when identifying tire wear status, the fusion weight of the local branch can be increased, and when identifying road surface type, the fusion weight of the global branch can be increased.

[0127] Among them, the global normalized features and local normalized features are fused according to the first fusion weight and the second fusion weight. Feature fusion can be achieved through a gating mechanism, as shown in formula (1).

[0128] Formula (1):

[0129] ;

[0130] in, It is the first fusion weight of the global branch. It is a globally normalized feature; It is the second fusion weight of the local branch. It is a local normalized feature. It is an activation function (Sigmoid function). It is element-wise multiplication.

[0131] It should be noted that the relevant technologies face two challenges. One is that in video features, since the tire surface only occupies a small part, but contains key wear information, high-precision local feature extraction is required. The other is that road type recognition requires global contextual information to understand the overall features of the terrain. Traditional attention mechanism networks have obvious limitations in handling these tasks: channel attention mechanisms are difficult to capture spatial details, spatial attention mechanisms have high computational complexity, and are not sensitive enough to small targets.

[0132] In the above embodiments, the local branch is used to extract spatial detail features, further capturing small-scale targets and boundary accuracy, while the global branch uses a lightweight self-attention mechanism to address the limited receptive field of convolution, thereby establishing long-distance spatial dependencies. Feature fusion of local and global enhancement features ensures the complementary advantages of the local and global branches, achieving efficient fusion of local details and global context. While maintaining computational efficiency, it significantly improves the ability to extract key features of the tire contact area, making it suitable for handling subtle visual changes in tire-road interaction under complex terrain. In real-vehicle testing, for driving on complex terrain, the fusion of local and global enhancement features significantly improves the accuracy of multi-modal fusion and multi-task processing.

[0133] In a specific example, refer to Figure 9 The multi-task model, which integrates multimodal fusion, processes forward-looking video streams, vehicle state data, and tire-road interaction data, including:

[0134] For the forward-looking video stream, vehicle status data, and tire-road interaction data acquired over a period of time, the forward-looking video stream includes multiple video frames, such as video frames with a size of 1280x720; the fused signal data is determined based on the vehicle status data and tire-road interaction data; for example, the fused signal data includes: (longitudinal acceleration, lateral acceleration, vertical acceleration, vehicle speed, tire pressure, tire temperature, electrical signal, time domain features, and frequency domain features).

[0135] The forward-looking video stream is input into the visual feature extraction submodule to obtain the initial video feature feature1. The visual feature extraction submodule includes a convolutional layer and five cascaded visual feature extraction units (including CBS network and C2F network).

[0136] The signal flow feature extraction module integrates the input signal data to obtain signal feature feature2. The signal flow feature extraction module includes three cascaded signal feature extraction sub-modules (including cascaded fully connected layers, batch normalization layers, activation function layers, and output layers).

[0137] The initial video feature 1 is processed by the spatiotemporal feature extraction submodule to obtain the temporal synchronization feature 3;

[0138] The initial video feature 1 is processed by the ED-CAFM module to obtain the enhanced video feature 4;

[0139] Signal feature 2 and enhanced video feature 4 are fused through a concat layer to obtain multimodal fusion feature 5;

[0140] Convolutional processing is performed on temporal synchronization feature 3 to obtain feature 6, convolutional processing is performed on enhanced video feature 4 to obtain feature 7, and feature 6 and feature 7 are fused by concat processing to obtain fused video feature feature 8.

[0141] The fused video feature feature8 is obtained by upsampling, convolution, and upsampling processing to obtain the target video feature feature9;

[0142] The multimodal fusion feature 5 is processed by convolution to obtain feature 10;

[0143] feature10 is processed by average pooling and fully connected layers to obtain the target road surface feature feature11;

[0144] feature10 is processed through a fully connected layer to obtain the target tire feature feature12;

[0145] The drivable region is obtained by processing the target video feature9 through the drivable region segmentation module;

[0146] The road surface detection module processes the target road surface feature11 to obtain the road surface type;

[0147] The tire wear status is obtained by processing the target tire feature12 through the tire detection module.

[0148] In the above embodiments, the problem of temporal asynchrony between the forward-looking video stream and vehicle state data and tire road surface interaction data is solved by spatiotemporal feature extraction. Multimodal fusion features are obtained through feature fusion, which have rich fusion information of vision and signal, providing high-quality input for subsequent task processing. The three tasks of image segmentation, road surface detection and tire wear detection are decoupled, and the most suitable feature input is provided for each task, realizing efficient collaboration of multiple tasks.

[0149] In some embodiments, determining the terrain risk value based on the drivable area and road surface type includes: a basic risk value corresponding to the road surface type; determining a traffic risk value based on the drivable area; determining a dynamic adjustment value based on the vehicle's dynamic data; and determining the terrain risk value based on the basic risk value, the traffic risk value, and the dynamic adjustment value.

[0150] Among them, the basic risk value represents the passability risk corresponding to the road surface type. For example, dry cement road surface is easy to pass and its corresponding basic risk value is low; wet and slippery road surface may slip and its corresponding basic risk value is medium; and road surface with heavy snow accumulation may be difficult to pass and its corresponding basic risk value is high.

[0151] The traffic risk value represents the risk of insufficient drivable area. For example, if the drivable area is large or the vehicle is far from the boundary of the drivable area, the drivable area is sufficient and the traffic risk value is low. Conversely, if the drivable area is small or the vehicle is close to the boundary of the drivable area, the drivable area is insufficient and the traffic risk value is high.

[0152] The dynamic adjustment value represents the risk changes caused by the vehicle's motion state; for example, a vehicle speed greater than the preset speed leads to increased risk, and a steering angle greater than the preset steering angle leads to increased risk; conversely, a vehicle speed not greater than the preset speed leads to decreased risk, and a steering angle not greater than the preset steering angle leads to increased risk.

[0153] Among them, the terrain risk value reflects the risk of a vehicle traveling in the terrain formed by the current drivable area and road surface type.

[0154] For example, a high terrain risk value indicates a higher risk of vehicle skidding, difficulty getting out of trouble, tire overheating, and insufficient feasible area; a low terrain risk value indicates a lower risk of vehicle skidding, difficulty getting out of trouble, tire overheating, and insufficient feasible area; for example, the terrain risk value is a value between 0 and 1.

[0155] Specifically, the basic risk value corresponding to each road surface type is determined based on a preset risk mapping table. The preset risk mapping table is determined based on expert knowledge and includes the basic risk values ​​corresponding to various road surface types.

[0156] For example, the basic risk value for ordinary asphalt pavement and dry cement pavement is a low risk value, such as 0.1-0.3; the basic risk value for wet and slippery pavement, gravel pavement, and lightly snow-covered pavement is a medium risk value, such as 0.4-0.6; and the basic risk value for loose sand pavement, muddy pavement, icy pavement, and heavily snow-covered pavement is a high risk value, such as 0.7-0.9.

[0157] Specifically, a first distance is determined between the vehicle and the left boundary of the drivable area, a second distance is determined between the vehicle and the right boundary of the drivable area, and a third distance is determined between the vehicle and the front boundary of the drivable area. The minimum distance between the first and second distances is taken as the lateral safety distance, and the third distance is taken as the longitudinal safety distance. The lateral and longitudinal safety distances are normalized to obtain the normalized lateral safety distance and the normalized longitudinal safety distance. The average of the normalized lateral and longitudinal safety distances is then used to obtain the normalized safety distance.

[0158] The area of ​​the drivable zone is used as the passage area, and the passage area is normalized to obtain the normalized safe area.

[0159] The normalized traffic area and normalized safety distance are weighted and summed to obtain the drivable safety value; the difference between 1 and the drivable safety value is calculated to obtain the traffic risk value; the weights of the normalized traffic area and normalized safety distance can be set according to actual needs.

[0160] In other words, the larger the drivable area and the greater the safe distance between the vehicle and the boundary of the drivable area, the lower the traffic risk value.

[0161] Optionally, if the passage area is greater than the preset area and the normalized safety distance is greater than the preset distance, then the passage risk value is determined to be 0.

[0162] Specifically, taking vehicle dynamic data including vehicle speed, steering angle, and gradient as examples, if the vehicle speed is greater than a preset speed threshold and the basic risk value corresponding to the road type is a high-risk value, then the first preset value is used as the speed risk adjustment value; otherwise, the speed risk adjustment value is determined to be 0. If the steering angle is greater than a preset steering angle threshold, it indicates that the vehicle is in a large steering angle condition, and the lateral adhesion requirement increases; then the second preset value is used as the steering risk adjustment value; otherwise, the steering risk adjustment value is determined to be 0. If the gradient is greater than a preset gradient threshold, such as when driving on an uphill or downhill road, then the third preset value is used as the gradient risk adjustment value; otherwise, the gradient risk adjustment value is determined to be 0. The sum of the speed risk adjustment value, steering risk adjustment value, and preset gradient adjustment value is used as the dynamic adjustment value.

[0163] The sum of the basic risk value, the access risk value, and the dynamic adjustment value is used as the terrain risk value.

[0164] In the above embodiments, the current terrain risk value is dynamically determined by integrating road surface type, drivable area and vehicle dynamic data. Compared with determining the terrain risk value based solely on road surface type, this improves the accuracy of the terrain risk value and makes the terrain risk value match the current actual driving situation.

[0165] In some embodiments, determining a tire-terrain coupling risk value based on tire wear condition and terrain risk value includes: determining a long-term tire condition value and a tire aging value based on the vehicle's historical driving data; determining a tire health value based on tire wear condition, tire aging value, and long-term tire condition value; and determining a tire-terrain coupling risk value based on the tire health value and terrain risk value.

[0166] Historical driving data includes, but is not limited to: mileage, average load, frequency of rapid acceleration, vehicle usage time, and high-temperature exposure data.

[0167] Among them, the tire aging value indicates the degree of tire aging; the tire long-term condition value is used to quantify the comprehensive evaluation value of the tire under the cumulative effects of multiple factors such as mileage and driving conditions during long-term use.

[0168] Specifically, historical driving data includes: mileage, average load, frequency of rapid acceleration, vehicle usage time, and high temperature exposure data; based on mileage, average load, and frequency of rapid acceleration, the long-term tire condition value is determined through a gradient boosting decision tree (GBDT).

[0169] In practical applications, the long-term tire status value can be updated once every preset mileage, for example, automatically updated every 1,000 kilometers. In addition, the gradient boosting decision tree can be optimized through online learning to improve the accuracy of the long-term tire status value.

[0170] Based on the mapping relationship, the time aging index corresponding to the vehicle's usage time is obtained. The mapping relationship represents the relationship between the vehicle's usage time and the time aging index, and the mapping relationship is predetermined.

[0171] The temperature aging index is determined based on the high temperature exposure data of the tires. For example, the high temperature exposure data includes the number of historical high temperature days, and the corresponding temperature aging index is obtained based on the number of historical high temperature days; or, the historical high temperature exposure days and the number of times the brakes were subjected to high temperature are combined and weighted to obtain the temperature aging index.

[0172] The tire aging value is obtained by weighted summation of the time aging index and the temperature aging index; for example, the tire aging value is determined according to formula (2).

[0173] Formula (2): ;

[0174] in, It is the weight of the time aging index. This is the weight of the temperature aging index; for example, the weight of the time aging index is 0.7, and the weight of the temperature aging index is 0.3.

[0175] In practical applications, tire aging values ​​can be updated once every preset mileage, for example, the tire aging value can be automatically updated every 1,000 kilometers.

[0176] Tire health value is determined based on a comprehensive assessment of the tire's short-term and long-term condition. A higher tire health value indicates a healthier overall tire condition, while a lower tire health value indicates a less healthy overall tire condition. Among these, tire wear condition is related to current road surface and vehicle dynamic data and is used to reflect the tire's short-term condition; tire aging value and tire long-term condition value are used to reflect the tire's long-term condition.

[0177] Specifically, since the tire wear condition is represented by the tire wear value, the tire wear value, tire aging value and tire long-term condition value are weighted and summed to obtain the tire health value. The weights of the tire wear value, tire aging value and tire long-term condition value can be set according to actual needs. For example, the tire health value is calculated by formula (3).

[0178] Formula (3):

[0179] ;

[0180] Among them, THI (Tire Health Index) is the tire health value; It is the weight of the tire wear value. It is the weight of the tire aging value. These are the weights of the tire's long-term condition values; for example, It is 0.6. It is 0.2. It is 0.2.

[0181] Optionally, the method further includes determining the remaining useful life (RUL) of the tire based on the tire's long-term condition value.

[0182] Specifically, the tire health value and terrain risk value are combined to obtain the tire-terrain coupling risk value. For example, the tire health value and terrain risk value are weighted and summed to obtain the tire-terrain coupling risk value; or the product of the tire health value and terrain risk value is used as the tire-terrain coupling risk value.

[0183] In the above embodiments, the tire health value is determined based on the tire wear condition, tire aging value, and tire long-term condition value. That is, the tire health status is determined by comprehensively considering the short-term wear and long-term condition of the tire, which improves the accuracy of the tire health value. The tire-terrain coupling risk value is determined by comprehensively considering the tire health value and the terrain risk value, which accurately reflects the risk situation of the dynamic interaction between the tire and the terrain. A dynamic and adaptive risk assessment system is constructed, which transforms the multimodal perception results into risk indicators with clear engineering significance, providing accurate decision-making basis for subsequent vehicle control, thereby improving the reliability of all-terrain intelligent driving.

[0184] In some embodiments, determining a tire-terrain coupling risk value based on tire health value and terrain risk value includes: determining a severe weather value based on the current weather conditions; and determining a tire-terrain coupling risk value based on tire health value, terrain risk value, and severe weather value.

[0185] Specifically, the corresponding severe weather value is obtained based on the current weather conditions. For example, if the weather conditions are rain, snow, or fog, the severe weather value is the first coefficient; if the weather conditions are not rain, snow, or fog, the severe weather value is the second coefficient; the second coefficient is less than the first coefficient; for example, the second coefficient is 0.

[0186] Optionally, a tire health discount factor is determined based on the tire health value, and the tire health value and the tire health value are negatively correlated. A tire health adjustment factor is determined based on the tire health discount factor. An adverse weather adjustment factor is determined based on the adverse weather value. Based on the terrain risk value, the tire health adjustment factor and the adverse weather adjustment factor are multiplied to obtain the tire terrain coupling risk value. For example, the tire terrain coupling risk value is calculated according to formula (4).

[0187] Formula (4): ;

[0188] Among them, CRI (Coupling Risk Index) is the tire-terrain coupled risk value; TRL (Terrain Risk Level) is the terrain risk value; the tire health discount factor is negatively correlated with the tire health value THI. It is the tire health adjustment factor, and WRI (Weather Risk Index) is the severe weather value; It is an adverse weather adjustment factor.

[0189] In the above embodiments, the tire terrain coupling risk value is determined based on weather conditions, tire health value, and terrain risk value. This takes into account the impact of severe weather on the current wear condition of the tire and its adaptability in the current terrain environment, thereby improving the accuracy of the tire terrain coupling risk value.

[0190] Optionally, the method further includes: determining the probability value of a preset risk based on the tire-terrain coupling risk value, wherein the preset risks include: slippage risk, overheating risk and abnormal wear risk.

[0191] Specifically, the tire-road friction coefficient corresponding to the road surface type is obtained, and the probability value of skidding risk is determined based on the tire-road friction coefficient, driving torque, and tire-terrain coupling risk value. For example, the tire-road friction coefficient, driving torque, and tire-terrain coupling risk value are input into the skidding risk prediction model to obtain the probability value of skidding risk.

[0192] Acquire driving condition data, including continuous driving time and speed; process the driving condition data using a rolling resistance model to obtain rolling resistance; determine tire temperature based on rolling resistance; and determine the probability value of overheating risk based on tire temperature and tire-terrain coupling risk value; for example, determine the probability value of overheating risk by looking up a table based on tire temperature and tire-terrain coupling risk value.

[0193] Obtain the road surface roughness corresponding to the road surface type, obtain the tire slip ratio and positioning parameters, including toe angle and camber angle; determine the positioning deviation based on the toe angle and camber angle, and determine the probability value of abnormal wear risk based on the road surface roughness, slip ratio, positioning deviation and tire terrain coupling risk value.

[0194] Optionally, a risk warning can be issued based on a preset probability value. For example, if the preset probability value is greater than a preset probability threshold, a risk warning can be issued based on the preset risk. For example, if the probability value of skidding risk is greater than the preset probability threshold, a skidding risk warning can be displayed on the vehicle's infotainment system to alert the driver. The predicted probability threshold can be set according to actual needs, and this application embodiment does not limit it.

[0195] In the above embodiments, based on the tire terrain coupling risk value, the probability values ​​of slippage risk, overheating risk and abnormal wear risk are predicted, and risk warnings are given according to the probability values, so that the driver can make driving adjustments based on the warnings. By giving early warnings, slippage, overheating and abnormal wear can be effectively reduced, tire life can be improved, and safety hazards such as tire blowouts can be effectively avoided.

[0196] Optionally, the method further includes: determining a comprehensive risk level based on the tire terrain coupling risk value and the tire health value, and issuing an early warning based on the comprehensive risk level.

[0197] Specifically, if the tire health value (THI) is in the first range (e.g., THI > 80) and the tire terrain coupling risk value (CRI) is in the second range (e.g., CRI < 0.4), then the overall risk level is determined to be Level 1, and no warning is issued.

[0198] If the tire health value (THI) belongs to the third interval (e.g., THI ∈ (70, 80]), or the tire terrain coupling risk value (CRI) belongs to the fourth interval (e.g., CRI ∈ [0.4, 0.6)), then the overall risk level is determined to be level two, and attention warning is issued through the first method. The first method can be a visual method, such as displaying a warning prompt on the vehicle's HMI interface.

[0199] If the tire health value (THI) belongs to the fifth interval (e.g., THI ∈ (50, 70]), or the tire terrain coupling risk value (CRI) belongs to the sixth interval (e.g., CRI ∈ [0.6, 0.8)), then the comprehensive risk level is determined to be level three, and a warning is issued through the second method. The second method can be an audio method, such as issuing a warning through voice.

[0200] If the tire health index (THI) is in the seventh range (e.g., THI ≤ 50), or the tire terrain coupling risk index (CRI) is in the eighth range (e.g., CRI ≥ 0.8), the overall risk level is determined to be level four, and a severe warning is issued through a third method. This third method can include visual and auditory methods. In addition, if the current conditions for intelligent driving activation are met, the intelligent driving function can be activated to actively intervene.

[0201] In practical applications, the comprehensive risk level and early warning status are packaged into a standardized risk assessment message, which is sent to the power system collaborative control module via the CAN bus to trigger the corresponding control strategy.

[0202] Optionally, a comprehensive risk level can be determined periodically based on the tire-terrain coupling risk value and the tire health value, and an early warning can be issued based on the comprehensive risk level. This period can be 100ms, so as to capture the dynamic changes of tire and terrain conditions in a timely manner and provide real-time decision support for vehicle control.

[0203] In the above embodiments, the severity is determined based on the comprehensive risk level, and graded early warnings are issued to provide a basis for subsequent control decisions.

[0204] In some embodiments, vehicle dynamic control based on tire-terrain coupling risk value includes: intelligently controlling any subsystem among multiple subsystems of the vehicle based on tire-terrain coupling risk value, wherein the multiple subsystems include a power subsystem, a chassis system, an energy recovery subsystem, and a thermal management subsystem.

[0205] Specifically, based on the tire-terrain coupling risk value, at least one of the power subsystem, chassis system, energy recovery subsystem, and thermal management subsystem is controlled.

[0206] Controlling the powertrain subsystem can involve adjusting the torque required for driving; controlling the chassis system can involve adjusting suspension parameters and tire pressure; controlling the energy recovery subsystem can involve adjusting the energy recovery torque; and controlling the thermal management subsystem can involve adjusting the battery cooling mode.

[0207] In the above embodiments, the tire terrain coupling risk value is converted into control commands to control a single subsystem of the vehicle or to control multiple subsystems as a whole; a closed-loop full-link system of multimodal perception, risk assessment and dynamic control is realized, breaking the data silos between tire detection and vehicle control, and achieving global optimization of safety, energy efficiency and component life through the collaborative optimization of multiple subsystems.

[0208] In some embodiments, based on tire-terrain coupling risk values, intelligent control is performed on any subsystem among multiple subsystems of the vehicle, including:

[0209] When any subsystem includes a powertrain subsystem, the driving torque demand is adjusted based on the tire-terrain coupling risk value and the drivable area; when any subsystem includes an energy recovery subsystem, the energy recovery torque is adjusted based on the tire-terrain coupling risk value; when any subsystem includes a chassis system, the suspension parameters and tire pressure are adjusted based on the tire-terrain coupling risk value; when any subsystem includes a thermal management subsystem, the battery cooling mode is adjusted based on the tire-terrain coupling risk value and the ambient temperature.

[0210] Specifically, determine the coupling risk level corresponding to the tire terrain coupling risk value, determine the passage risk value based on the drivable area, and determine the passage risk level based on the passage risk value; based on the coupling risk level and the passage risk level, adjust at least one of the following parameters: driving demand torque, energy recovery torque, suspension parameters, tire pressure, and battery cooling parameters.

[0211] For example, in the case where any subsystem includes a power subsystem, if the coupling risk level is low and the traffic risk level is low, the driver's required torque is not adjusted; if the coupling risk level is medium and the traffic risk level is low, the driver's required torque is limited according to a first ratio; if the coupling risk level is high or the traffic risk level is high, the driver's required torque is reduced according to a second ratio.

[0212] In the case where any subsystem includes an energy recovery subsystem, if the coupling risk level is low and the travel risk level is low, the energy recovery torque is set according to the strong recovery mode; if the coupling risk level is medium and the travel risk level is low, the energy recovery torque is set according to the medium recovery mode; if the coupling risk level is high or the travel risk level is high, the energy recovery torque is set according to the weak recovery mode.

[0213] In the case of any subsystem including the chassis recovery subsystem, if the coupling risk level is low and the travel risk level is low, the suspension parameters are set according to the comfort mode and the tire pressure is set according to the standard mode; if the coupling risk level is medium and the travel risk level is low, the suspension parameters are set according to the automatic mode and the standard tire pressure is fine-tuned; if the coupling risk level is high or the travel risk level is high, the suspension parameters are set according to the off-road mode and the tire pressure is automatically set according to the road surface type.

[0214] In the case where any subsystem includes a thermal management subsystem, the target driving torque is obtained by adjusting the driving torque based on the tire terrain coupling risk value and the drivable area. Based on the target driving torque and the ambient temperature, the battery thermal management and motor thermal management are controlled.

[0215] In the above embodiments, based on the coupling risk level and the traffic risk level, at least one of the driving demand torque, energy recovery torque, suspension parameters, tire pressure, and battery cooling parameters is dynamically adjusted, thereby achieving power distribution optimization, energy recovery efficiency optimization, and tire pressure adaptive optimization, which improves the passability of complex terrain and reduces the energy consumption of the vehicle under all-terrain conditions while ensuring safe driving.

[0216] In some embodiments, intelligent control of any subsystem among multiple subsystems of a vehicle based on tire-terrain coupling risk values ​​includes: determining a traffic risk value based on the drivable area; determining a comprehensive risk level based on the traffic risk value and the tire-terrain coupling risk value; obtaining the required torque limit parameter, regenerative torque gear, suspension mode, and tire pressure adjustment value corresponding to the comprehensive risk level; adjusting the driving required torque of the powertrain subsystem, the energy regenerative torque of the energy regeneration subsystem, the suspension parameters of the chassis system, and the tire pressure based on the required torque limit parameter, the regenerative torque gear, the suspension mode, and the tire pressure adjustment value; and adjusting the battery cooling mode of the thermal management subsystem based on the ambient temperature and the adjusted driving required torque.

[0217] Specifically, the process of determining the traffic risk value based on the drivable area can be found in the description in the above embodiments.

[0218] The traffic risk level is determined based on the range to which the traffic risk value belongs, and the coupling risk level is determined based on the tire terrain coupling risk value.

[0219] For example, when the traffic risk value belongs to the first traffic risk range, the traffic risk level is determined to be low risk; when the traffic risk value belongs to the second traffic risk range, the traffic risk level is determined to be high risk. The first and second traffic risk ranges can be set according to actual needs.

[0220] For example, when the tire-terrain coupling risk value belongs to the first coupling interval (e.g., CRI < 0.4), the coupling risk level is determined to be low risk; when the tire-terrain coupling risk value belongs to the second coupling interval (e.g., 0.4 ≤ CRI < 0.7), the coupling risk level is determined to be medium risk; when the tire-terrain coupling risk value belongs to the third coupling interval (e.g., CRI ≥ 0.7), the coupling risk level is determined to be high risk.

[0221] When the coupling risk level is low and the traffic risk level is low, the overall risk level is low. No adjustment is made to the driver's required torque, i.e., the driver's required torque is maintained. The suspension damping coefficient is adjusted to the first damping coefficient (for example, when the suspension is in comfort mode, the suspension damping coefficient K=0.7), the tire pressure is adjusted to the standard tire pressure (for example, 2.5 Bar), and the energy recovery torque is adjusted to the strong recovery torque (for example, -0.15g).

[0222] When the coupling risk level is medium risk and the traffic risk level is low risk, the overall risk level is medium risk. The driving torque demand is adjusted according to the first proportion, that is, the driving torque demand is smoothly controlled, for example, the driving torque demand is limited to 90% of the demand value (first proportion). The suspension damping coefficient is adjusted to the second damping coefficient (for example, the suspension is in automatic mode, the suspension damping coefficient K=0.85). The standard tire pressure is fine-tuned according to the road type (for example, the tire pressure adjustment value corresponding to the road type is ±0.2 Bar, and the tire pressure is fine-tuned according to ±0.2 Bar). The energy recovery torque is adjusted to the medium recovery torque (for example, -0.10g).

[0223] When the coupling risk level is high risk, or the passage risk level is high risk, the overall risk level is high risk. The driving torque demand is adjusted according to the second ratio, for example, the driving torque demand is limited to 70% of the demand value (second ratio). The suspension damping coefficient is adjusted to the third damping coefficient (for example, the suspension is in off-road mode, and the suspension damping coefficient K=1.0). The automatic tire pressure adjustment function is activated to automatically determine the tire pressure according to the road surface type (for example, in sand mode, the tire pressure is 1.8 Bar). The energy recovery torque is adjusted to weak recovery torque (for example, -0.05g).

[0224] After adjusting the driving torque based on the tire terrain coupling risk value and the drivable area, the target driving torque is obtained. Based on the target driving torque and the ambient temperature, the battery cooling mode is adjusted.

[0225] For example, if the target driving torque demand is high (high load) and the ambient temperature is high, the active cooling of the battery is activated; if the target driving torque demand is low (low load) and the ambient temperature is low, the cooling power of the battery is reduced to save energy.

[0226] Optionally, the method further includes: adjusting the motor thermal management according to the target driving demand torque; for example, determining the torque output curve based on the target driving demand torque over a period of time; if the torque output curve represents continuous high torque, then increasing the motor coolant flow rate; if the torque output curve represents intermittent low torque, then setting the motor to adopt an economic cooling mode.

[0227] Optionally, the method further includes: adjusting the driving demand torque according to the tire terrain coupling risk value and the drivable area to obtain the target driving demand torque; and adaptively adjusting the target driving demand torque based on the tire adhesion margin to obtain the actual output torque.

[0228] Specifically, the available adhesion is determined based on the estimated road surface friction coefficient and the vertical load; for example, the product of the estimated road surface friction coefficient and the vertical load is taken as the available adhesion. The required adhesion is determined based on the absolute values ​​of the lateral and longitudinal forces of the vehicle; for example, the sum of the absolute values ​​of the lateral and longitudinal forces is taken as the required adhesion. The tire adhesion margin is determined based on the available adhesion and the required adhesion; for example, the difference between the available adhesion and the required adhesion is calculated, and the ratio of this difference to the available adhesion is taken as the adhesion margin.

[0229] When the adhesion margin is less than the preset adhesion margin (e.g., 0.15), the adhesion margin and the preset adhesion margin are used as torque limiting coefficients. The minimum value is determined between the preset coefficient (e.g., 1) and the torque limiting coefficient. The product between the minimum value and the target driving torque requirement is used as the actual output torque.

[0230] In one alternative approach, the suspension damping coefficient is adjusted based on the coupling risk level and the traffic risk level to obtain the target damping coefficient; further, the suspension height and the target damping coefficient are adjusted based on the terrain risk value (TRL).

[0231] For example, if the terrain risk value (TRL) is high-risk terrain, the suspension height is increased by a first preset height value (e.g., 30mm) to increase the ground clearance of the chassis; if the terrain risk value (TRL) is low-risk terrain, the suspension height is decreased by a second preset height value (e.g., 15mm) to improve stability.

[0232] In one alternative approach, tire pressure is adaptively adjusted based on road surface type. For example, when the road surface is soft (such as sand or mud), the tire pressure is controlled at 1.6-1.8 Bar to increase the contact area and improve buoyancy; when the road surface is mixed (such as gravel or snow), the tire pressure is controlled at 2.0-2.2 Bar to balance passability and rolling resistance; and when the road surface is hard (such as paved roads), the tire pressure is controlled at 2.4-2.6 Bar to optimize energy efficiency and handling.

[0233] In one alternative approach, suspension parameters are automatically adjusted based on the driver's steering intention; for example, if the steering angular velocity is greater than a preset steering angular velocity (e.g., ... If the tire-terrain coupling risk value (CRI) increases during steering, the torque will be fine-tuned, for example, by reducing the torque required for driving, to assist steering.

[0234] In one alternative approach, the energy recovery intensity can be adjusted based on the road surface type. For example, if the road surface is low-traction, the energy recovery intensity can be reduced to avoid tire lock-up. In addition, the recovery strategy can be adjusted in advance using navigation data.

[0235] In one alternative approach, the all-terrain mode is switched based on the tire terrain coupling risk value (CRI). For example, when the tire terrain coupling risk value is in the fourth coupling range (e.g., CRI < 0.3), corresponding to highway driving scenarios, the mode is switched to economy mode; when the tire terrain coupling risk value is in the fifth coupling range (e.g., 0.3 ≤ CRI < 0.6), corresponding to mixed terrain driving scenarios, the mode is switched to automatic mode; when the tire terrain coupling risk value is in the sixth coupling range (e.g., 0.6 ≤ CRI < 0.8), corresponding to complex terrain driving scenarios, the mode is switched to off-road mode; and when the tire terrain coupling risk value is in the seventh coupling range (e.g., CRI ≥ 0.8), corresponding to driving scenarios requiring traction, the mode is switched to traction-avoidance mode.

[0236] In some embodiments, after performing vehicle dynamic control based on tire terrain coupling risk value, the method further includes: determining control effect evaluation index according to the current system state of the vehicle; adjusting the parameters of the control strategy according to the control effect evaluation index and preset index threshold; the control strategy is the strategy used to perform vehicle dynamic control.

[0237] The current system status of the vehicle includes: wheel speed, vehicle speed, steering angle, yaw rate, lateral acceleration, vertical acceleration, regenerative electrical power, regenerative mechanical power, total battery current, total battery voltage, high-voltage side power, and tire temperature time-series data.

[0238] The evaluation indicators for control effectiveness are determined based on a comprehensive assessment of safety, economic, comfort, and durability indicators.

[0239] Specifically, the tire slip ratio is calculated based on wheel speed; the stability variance is calculated based on vehicle speed, steering angle, yaw rate, and lateral acceleration; and the safety index is determined based on the tire slip ratio and stability variance. For example, the tire slip ratio and stability variance are normalized separately, and the normalized tire slip ratio and stability variance are weighted and summed to obtain the safety index.

[0240] The energy recovery efficiency is determined based on the recovered electrical power and recoverable mechanical power; the total power consumption of the system is determined based on the total battery current, total battery voltage, and high-voltage side power; and economic indicators are determined based on the energy recovery efficiency and total power consumption of the system. For example, the energy recovery efficiency and total power consumption of the system are scored separately, and the economic indicators are obtained by weighted summation of the energy recovery efficiency score and the total power consumption score.

[0241] Comfort indicators are determined based on vertical acceleration, such as identifying the vertical acceleration range to which the vertical acceleration belongs and using the preset score corresponding to the vertical acceleration range as the comfort indicator; tire temperature change rate is determined based on tire temperature time series data; durability indicators are determined based on tire temperature change rate and tire wear condition, such as identifying the change rate range to which the tire temperature change rate belongs and using the preset score corresponding to the change rate range as the tire temperature score, and then weighted summing the tire wear condition (tire wear value) and the tire temperature score to obtain the durability indicator.

[0242] The control effect evaluation index is obtained by weighting and summing the safety index, economic index, comfort index and durability index, as shown in formula (5).

[0243] Formula (5): ;

[0244] in, It is an indicator for evaluating the effectiveness of control. It is a safety indicator. It is the weight of the security indicator. It is an economic indicator. It is the weight of economic indicators. It is a comfort indicator. It is the weight of comfort indicators. It is a durability indicator. It is the weight of the durability index; , , , It can be set according to actual needs.

[0245] If the control effectiveness evaluation index is less than the preset threshold, then based on the evaluated tire-terrain coupling risk value, the vehicle undergoes dynamic control as a negative control case. The control strategy is then adjusted based on these negative control cases to optimize it. For example, a reinforcement learning algorithm can be used to optimize the control strategy, continuously improving the accuracy and adaptability of decisions through ongoing learning.

[0246] If the control effect evaluation index is not less than the preset index threshold, then based on the evaluation: the vehicle dynamic control is performed on the whole vehicle based on the tire terrain coupling risk value, and the positive sample of the control case is put into the strategy library.

[0247] In the above embodiments, after performing whole-vehicle dynamic control based on the tire terrain coupling risk value, the control effect is evaluated, and the control strategy is corrected based on the control effect evaluation index, realizing feedforward and feedback composite control. A complete closed-loop decision-making process is formed through perception, control, evaluation and adjustment, and the reliability of all-terrain intelligent driving is continuously improved through optimization.

[0248] In one alternative approach, the vehicle undergoes fault detection, the fault level is determined based on the fault detection results, and fault handling is performed according to the fault level.

[0249] Specifically, fault detection is performed through sensor data validity verification, control command rationality check, and system function integrity diagnosis. If a sensor abnormality is detected, the fault level is determined to be a level one fault, and the sensor estimate is used to replace the actual value detected by the sensor.

[0250] Optionally, if a visual sensor malfunction is detected, such as a malfunction of the forward-facing camera, vehicle control is performed based on data from other sensors, and performance is degraded, such as limiting the maximum vehicle speed or degrading vision-related intelligent driving functions.

[0251] If an actuator malfunction is detected and the fault level is determined to be a level 2 fault, functional reconfiguration is performed to ensure that core functions are maintained; for example, control tasks are performed by redundant actuators, or the control is switched from independent control to differential assistance, to ensure basic driving capabilities of longitudinal control, lateral control, and stability control.

[0252] If a system-level anomaly is detected, and the fault level is determined to be a level three fault, a safe shutdown is initiated, and manual intervention is requested.

[0253] Optionally, if the risk assessment exceeds the time limit, such as if the tire-terrain coupling risk value is not obtained within a preset time period, a conservative control strategy is implemented, such as reducing the maximum vehicle speed, increasing safety redundancy, or disabling some intelligent driving functions, in order to improve safety.

[0254] In the above embodiments, vehicle fault detection and fault handling are performed, enabling the vehicle to perform safe driving even in the event of an anomaly, thereby improving the reliability and safety of vehicle control.

[0255] The vehicle control method based on multimodal data provided in this application integrates forward-looking video streams, vehicle state data, and tire-road interaction data through multimodal fusion and multi-task processing to obtain drivable area, road surface type, and tire wear status. This achieves dynamic correlation between drivable area, road surface type, and tire wear status, avoiding the separation of tire wear status from other sensing data and improving the accuracy of drivable area, road surface type, and tire wear status. Based on the drivable area, road surface type, and tire wear status, a tire-terrain coupling risk value is determined. Since the tire is the only contact point between the vehicle and the road surface, the tire status has a significant impact on driving safety and passability. In complex terrain, the tire-terrain coupling risk value can accurately reflect the risk of dynamic interaction between the tire and the terrain. Based on the tire-terrain coupling risk value, the vehicle is dynamically controlled, breaking down the data silos between tire detection and vehicle control, improving driving safety and passability in complex terrain, and enhancing the reliability of all-terrain intelligent driving.

[0256] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0257] Figure 10 A schematic diagram of the vehicle cooperative control system provided in this application, such as Figure 10 As shown, the vehicle cooperative control system includes: a control component and multiple subsystems; the control component is used to execute the above-mentioned vehicle control method based on multimodal data; the control component interacts with multiple subsystems of the vehicle to realize dynamic control of the entire vehicle.

[0258] The control components adopt a layered architecture. The perception layer is used to acquire multimodal perception data and perform multimodal fusion multi-task processing. The cognition layer is used to perform risk assessment. The decision layer is used to generate control strategies based on the risk assessment results. The execution layer is used to execute the control strategies. The verification layer is used to detect the control effect and form a closed-loop optimization.

[0259] Specifically, refer to Figure 11 The control components include: a multimodal perception module, a multimodal multitasking parsing module, a tire terrain risk assessment module, an all-terrain collaborative control module, and a closed-loop optimization module.

[0260] The multimodal perception module is used to acquire multimodal perception data, specifically: acquiring the forward-looking video stream captured by the forward-looking camera; acquiring tire road surface action data collected by the tire sensor; and acquiring vehicle status data collected by the vehicle status sensor. Among these, the vehicle status data collected by the vehicle status sensor includes: triaxial acceleration and vehicle speed acquired by the triaxial acceleration sensor, and tire pressure and tire temperature acquired by the TPMS system.

[0261] The multimodal multitasking parsing module is used to perform multimodal fusion and multitasking processing on multimodal perception data through the multimodal fusion multitasking model VS-MTF to obtain drivable area, road surface type and tire wear status.

[0262] The tire terrain risk assessment module is used to perform tire health assessment and terrain risk assessment based on drivable area, road surface type and tire wear condition through risk assessment engine, and output tire health value, terrain risk value and tire terrain coupling risk value; it is also used to provide risk warning based on tire terrain coupling risk value.

[0263] For example, the inputs to the tire terrain risk assessment module include: drivable area, road surface type, and tire wear condition, as well as: vehicle dynamic data, such as: vehicle speed, three-axis acceleration, steering angle, drive torque, braking status, slope, ambient temperature, and system running time; the outputs of the tire terrain risk assessment module include: tire health value (THI), terrain risk value (TRL), and tire terrain coupling risk value (CRI), as well as: detected preset risks, such as slippage risk, overheating risk, and abnormal wear risk.

[0264] The all-terrain cooperative control module is used to dynamically control the powertrain, chassis, energy recovery, and thermal management subsystems based on tire terrain coupling risk values ​​through the cooperative control engine. This includes torque distribution optimization, active suspension adjustment, tire pressure adaptive adjustment, and battery cooling mode adjustment.

[0265] For example, the inputs of the all-terrain cooperative control module include: risk assessment messages (including: tire health value THI, terrain risk value TRL, and tire-terrain coupling risk value CRI), driver intent messages (including accelerator pedal opening, brake pressure, steering angle, and gear selection), and system status messages (including battery SOC, motor temperature, and subsystem readiness status). The outputs of the all-terrain cooperative control module include multiple control command streams, which are sent to the powertrain subsystem, chassis system, energy recovery subsystem, and thermal management subsystem, respectively. The chassis system includes: suspension subsystem and tire pressure regulation subsystem.

[0266] The closed-loop optimization module is used to determine the control effect evaluation index after the execution of coordinated control. The control strategy is optimized based on the control effect evaluation index through a multi-objective optimization engine. The optimization objectives include improving safety, economy, comfort and durability.

[0267] Optionally, the operation of the control component is guided by an intelligent decision-making state machine, and the control component guides the intelligent decision-making state machine to switch states through state feedback.

[0268] Specifically, during the system self-test and sensor calibration phases, the system switches to the initialization state. When the tire-terrain coupling risk value corresponds to a low-risk scenario, economy is prioritized, so the system switches to the standard cruise state. When the tire-terrain coupling risk value corresponds to a medium-risk scenario, passability is prioritized, so the system switches to the terrain adaptation state. When the tire-terrain coupling risk value corresponds to a high-risk scenario, safety is prioritized, so the system switches to the risk avoidance state. When there is a need to get out of trouble, power needs to be maximized, so the system switches to the performance limit state. When a system fault is detected, the system performs functional degradation, so the system switches to the fault handling state. After executing collaborative control, when data collection and decision optimization are performed, the system switches to the learning optimization state.

[0269] For example, when the intelligent decision-making state machine is in standard cruise mode, if the tire terrain coupling risk value rises from 0.3 to above 0.7 and the vehicle speed is greater than the safe speed threshold, it switches to risk avoidance mode; when the intelligent decision-making state machine is in risk avoidance mode, if it detects that the tire terrain coupling risk value has dropped to below 0.2 and the duration has reached the set duration (e.g., 5 minutes) and the driving torque demand is stable, it switches to standard cruise mode.

[0270] The vehicle cooperative control system provided in this application implements a vehicle control method based on multimodal data through a control component. The control component interacts with multiple subsystems of the vehicle to achieve dynamic control of the whole vehicle. This avoids the separation of tire wear status from other sensing data, breaks down the data silos between tire detection and vehicle control, improves driving safety and passability in complex terrain, and enhances the reliability of all-terrain intelligent driving.

[0271] Figure 12 A schematic diagram of the structure of the vehicle control device based on multimodal data provided in this application is shown below. Figure 12 As shown, the vehicle control device 12 based on multimodal data provided in this embodiment includes:

[0272] The acquisition unit 1201 is used to acquire multimodal perception data of the vehicle, which includes forward-looking video stream, vehicle status data and tire-road interaction data.

[0273] The multi-task processing unit 1202 is used to perform multi-modal fusion processing based on forward-looking video stream, vehicle status data and tire road surface action data to obtain the drivable area, road surface type and tire wear status of the vehicle corresponding to the multi-modal perception data.

[0274] Terrain risk determination unit 1203 is used to determine the terrain risk value based on the drivable area and road surface type;

[0275] The coupling risk determination unit 1204 is used to determine the tire-terrain coupling risk value based on the tire wear condition and the terrain risk value.

[0276] The dynamic control unit 1205 is used to perform whole-vehicle dynamic control based on tire terrain coupling risk values.

[0277] In some embodiments, the multi-task processing unit 1202 is configured to perform multi-attention feature extraction and spatiotemporal feature extraction on the forward-looking video stream to obtain enhanced video features and temporal synchronization features; perform feature extraction based on vehicle state data and tire-road interaction data to obtain signal features; perform feature fusion on the enhanced video features and signal features to obtain multimodal fusion features; perform image segmentation processing based on the enhanced video features and temporal synchronization features to obtain drivable areas; and perform road surface detection and tire wear detection based on the multimodal fusion features to obtain road surface type and tire wear status.

[0278] In some embodiments, the multi-task processing unit 1202 is used to extract features from the forward-looking video stream to obtain initial video features; to perform dual-channel feature fusion of channel attention and spatial attention on the initial video features to obtain enhanced video features; and to extract spatiotemporal features from the initial video features to obtain temporal synchronization features.

[0279] In some embodiments, the multi-task processing unit 1202 is used to perform dual-channel feature fusion of channel attention and spatial attention on the initial video features to obtain enhanced video features, including: performing enhanced downsampling processing on the initial video features to obtain local enhanced features; performing convolutional attention processing on the initial video features to obtain global enhanced features; and performing feature fusion on the local enhanced features and global enhanced features to obtain enhanced video features.

[0280] In some embodiments, the terrain risk determination unit 1203 is used to obtain a basic risk value corresponding to the road surface type; determine a traffic risk value based on the drivable area; determine a dynamic adjustment value based on the vehicle's dynamic data; and determine a terrain risk value based on the basic risk value, the traffic risk value, and the dynamic adjustment value.

[0281] In some embodiments, the coupling risk determination unit 1204 is configured to determine the long-term tire condition value and tire aging value based on the vehicle's historical driving data; determine the tire health value based on the tire wear condition, tire aging value, and long-term tire condition value; and determine the tire-terrain coupling risk value based on the tire health value and the terrain risk value.

[0282] In some embodiments, the coupling risk determination unit 1204 is used to determine the severe weather value based on the current weather conditions; and to determine the tire-terrain coupling risk value based on the tire health value, the terrain risk value, and the severe weather value.

[0283] In some embodiments, the dynamic control unit 1205 is used to intelligently control any one of the multiple subsystems of the vehicle based on the tire-terrain coupling risk value, the multiple subsystems including the power subsystem, chassis system, energy recovery subsystem and thermal management subsystem.

[0284] In some embodiments, the dynamic control unit 1205 is configured to adjust the driving demand torque based on the tire-terrain coupling risk value and the drivable area when any subsystem includes a powertrain subsystem; adjust the energy recovery torque based on the tire-terrain coupling risk value when any subsystem includes an energy recovery subsystem; adjust the suspension parameters and tire pressure based on the tire-terrain coupling risk value when any subsystem includes a chassis system; and adjust the battery cooling mode based on the tire-terrain coupling risk value and the ambient temperature when any subsystem includes a thermal management subsystem.

[0285] In some embodiments, the dynamic control unit 1205 is configured to: determine a traffic risk value based on the drivable area; determine a comprehensive risk level based on the traffic risk value and the tire-terrain coupling risk value; acquire the required torque limit parameter, regenerative torque gear, suspension mode, and tire pressure adjustment value corresponding to the comprehensive risk level; adjust the driving demand torque of the powertrain subsystem, the energy recovery torque of the energy recovery subsystem, the suspension parameters of the chassis system, and the tire pressure based on the required torque limit parameter, the regenerative torque gear, the suspension mode, and the tire pressure adjustment value; and adjust the battery cooling mode of the thermal management subsystem based on the ambient temperature and the adjusted driving demand torque.

[0286] In some embodiments, the vehicle control device based on multimodal data further includes: a closed-loop optimization unit, used to determine a control effect evaluation index based on the current system state of the vehicle after performing whole-vehicle dynamic control based on the tire-terrain coupling risk value; and to adjust the parameters of the control strategy based on the control effect evaluation index and a preset index threshold; the control strategy is the strategy used to perform whole-vehicle dynamic control on the vehicle.

[0287] The vehicle control device based on multimodal data provided in this embodiment can execute the vehicle control method based on multimodal data provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0288] Figure 13 This is a structural diagram of the vehicle provided in this application. Figure 13 As shown, the vehicle 130 provided in this embodiment includes at least one processor 1301 and a memory 1302. Optionally, the device 130 also includes a communication component 1303. The processor 1301, memory 1302, and communication component 1303 are connected via a bus.

[0289] In a specific implementation, at least one processor 1301 executes computer execution instructions stored in memory 1302, causing at least one processor 1301 to perform the above-described method.

[0290] The specific implementation process of processor 1301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0291] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0292] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0293] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0294] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0295] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0296] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0297] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0298] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0299] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0300] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0301] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0302] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0303] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A vehicle control method based on multimodal data, characterized in that, include: Acquire multimodal perception data of the vehicle, the multimodal perception data including forward-looking video stream, vehicle status data and tire-road interaction data; Multi-attention feature extraction and spatiotemporal feature extraction are performed on the forward-looking video stream to obtain enhanced video features and temporal synchronization features; Based on the vehicle state data and the tire-road interaction data, feature extraction is performed to obtain signal features; The enhanced video features and the signal features are fused to obtain multimodal fused features; Image segmentation is performed based on the enhanced video features and the temporal synchronization features to obtain the drivable area; Based on the multimodal fusion features, road surface detection and tire wear detection are performed to obtain the road surface type and tire wear status; The terrain risk value is determined based on the drivable area and the road surface type; Based on the vehicle's historical driving data, the long-term tire condition value and tire aging value are determined. Based on the tire wear condition, tire aging value, and long-term tire condition value, the tire health value is determined. Based on the tire health value and the terrain risk value, the tire terrain coupling risk value is determined. The tire terrain coupling risk value is used to reflect the dynamic interactive risk of the tire operating in the terrain environment consisting of the current road surface type and drivable area with the current wear condition. Based on the tire-terrain coupling risk value, the vehicle is subjected to dynamic control.

2. The method according to claim 1, characterized in that, The step of performing multi-attention feature extraction and spatiotemporal feature extraction on the forward-looking video stream to obtain enhanced video features and temporal synchronization features includes: Feature extraction is performed on the forward-looking video stream to obtain initial video features; The initial video features are fused using dual-channel feature fusion with channel attention and spatial attention to obtain enhanced video features; Spatiotemporal feature extraction is performed on the initial video features to obtain temporal synchronization features.

3. The method according to claim 2, characterized in that, The process of fusing the initial video features with both channel attention and spatial attention to obtain enhanced video features includes: The initial video features are subjected to enhanced downsampling processing to obtain local enhanced features; The initial video features are subjected to convolutional attention processing to obtain global enhanced features; The local enhancement features and the global enhancement features are fused to obtain enhanced video features.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the terrain risk value based on the drivable area and the road surface type includes: Obtain the basic risk value corresponding to the road surface type; Determine the traffic risk value based on the drivable area; The dynamic adjustment value is determined based on the vehicle's dynamic data; The terrain risk value is determined based on the basic risk value, the access risk value, and the dynamic adjustment value.

5. The method according to any one of claims 1 to 3, characterized in that, The determination of the tire-terrain coupling risk value based on the tire health value and the terrain risk value includes: Determine the severity of severe weather based on the current weather conditions; The tire-terrain coupling risk value is determined based on the tire health value, the terrain risk value, and the severe weather value.

6. The method according to any one of claims 1 to 3, characterized in that, The process of performing vehicle dynamic control based on the tire terrain coupling risk value includes: Based on the tire-terrain coupling risk value, intelligent control is applied to any one of the multiple subsystems of the vehicle, including the powertrain subsystem, chassis system, energy recovery subsystem, and thermal management subsystem.

7. The method according to claim 6, characterized in that, The method of intelligently controlling any subsystem among multiple subsystems of the vehicle based on the tire terrain coupling risk value includes: In the case where any subsystem includes a powertrain subsystem, the driving torque demand is adjusted based on the tire-terrain coupling risk value and the drivable area; In the case where any subsystem includes an energy recovery subsystem, the energy recovery torque is adjusted according to the tire-terrain coupling risk value; In the case where any subsystem includes a chassis system, the suspension parameters and tire pressure are adjusted based on the tire-terrain coupling risk value; In the case where any subsystem includes a thermal management subsystem, the battery cooling mode is adjusted based on the tire terrain coupling risk value and ambient temperature.

8. The method according to claim 6, characterized in that, The method of intelligently controlling any subsystem among multiple subsystems of the vehicle based on the tire terrain coupling risk value includes: Determine the traffic risk value based on the drivable area; The overall risk level is determined based on the passage risk value and the tire terrain coupling risk value; Obtain the required torque limit parameters, recovery torque level, suspension mode, and tire pressure adjustment value corresponding to the comprehensive risk level; Based on the required torque limit parameter, the regenerative torque level, the suspension mode, and the tire pressure adjustment value, the driving required torque of the power subsystem, the energy regenerative torque of the energy regeneration subsystem, the suspension parameters of the chassis system, and the tire pressure are adjusted respectively. The battery cooling mode of the thermal management subsystem is adjusted based on the ambient temperature and the adjusted driving torque requirements.

9. The method according to any one of claims 1 to 3, characterized in that, After performing vehicle dynamic control based on the tire terrain coupling risk value, the method further includes: Determine the control effectiveness evaluation indicators based on the current system status of the vehicle; The parameters of the control strategy are adjusted based on the control effect evaluation index and the preset index threshold; the control strategy is the strategy used to perform dynamic control of the vehicle.

10. A vehicle cooperative control system, characterized in that, The system includes: a control component and multiple subsystems; The control component is used to execute the vehicle control method based on multimodal data as described in any one of claims 1 to 9; the control component interacts with the multiple subsystems of the vehicle to realize dynamic control of the entire vehicle.

11. A vehicle control device based on multimodal data, characterized in that, The device includes: The acquisition unit is used to acquire multimodal perception data of the vehicle, which includes forward-looking video stream, vehicle status data, and tire-road interaction data. The multi-task processing unit is used to perform multi-attention feature extraction and spatiotemporal feature extraction on the forward-looking video stream to obtain enhanced video features and temporal synchronization features; to perform feature extraction based on the vehicle state data and the tire-road interaction data to obtain signal features; to perform feature fusion on the enhanced video features and the signal features to obtain multimodal fusion features; to perform image segmentation processing based on the enhanced video features and the temporal synchronization features to obtain drivable areas; and to perform road surface detection and tire wear detection based on the multimodal fusion features to obtain road surface type and tire wear status. A terrain risk determination unit is used to determine a terrain risk value based on the drivable area and the road surface type; The coupling risk determination unit is used to determine the long-term tire condition value and tire aging value based on the vehicle's historical driving data, determine the tire health value based on the tire wear condition, the tire aging value and the long-term tire condition value, and determine the tire terrain coupling risk value based on the tire health value and the terrain risk value. The tire terrain coupling risk value is used to reflect the dynamic interactive risk of the tire operating in the terrain environment consisting of the current road surface type and drivable area with the current wear condition. A dynamic control unit is used to perform overall vehicle dynamic control based on the tire terrain coupling risk value.

12. A vehicle, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, Includes computer execution instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 9.

Images