Vehicle control system and method, electronic equipment, storage medium and vehicle

Abstract

The invention provides a vehicle control system and method, electronic equipment, a storage medium and a vehicle, relates to the technical field of vehicles, and is used for realizing road surface recognition. The system comprises a controller and a plurality of sensors connected with the controller, wherein the plurality of sensors at least comprise a tire noise sensor; the controller is configured to: receive sensor data sent by a plurality of sensors; and determining a road surface recognition result of a road on which the vehicle travels based on the sensor data sent by the various sensors.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more particularly to a vehicle control system, method, electronic device, storage medium, and vehicle. Background Technology

[0002] As the need for vehicles to navigate complex scenarios increases, these scenarios place higher demands on vehicle performance. For example, when driving on uneven roads, vehicles are affected by the bumps and undulations, impacting both vehicle adaptability and user comfort. To address this issue, identifying the road surface type and matching it with an appropriate driving mode can improve vehicle adaptability and driving experience in complex scenarios. Therefore, achieving road surface recognition is a pressing problem that needs to be solved. Summary of the Invention

[0003] The purpose of this application is to provide a vehicle control system, method, electronic device, storage medium, and vehicle, which aims to solve the problem of how to achieve road surface recognition.

[0004] In a first aspect, a vehicle control system is provided, including: a controller and a plurality of sensors connected to the controller, the plurality of sensors including at least: a tire noise sensor; the controller is configured to: receive sensor data sent by the plurality of sensors; and determine the road surface identification result of the road surface on which the vehicle is traveling based on the sensor data sent by the plurality of sensors.

[0005] The vehicle control system provided in this application embodiment can collect sensor data of the vehicle while driving through at least the tire noise sensor. The controller can identify the road surface of the road on which the vehicle is driving based on the sensor data of the tire noise sensor, thereby providing a basis for the vehicle to make intelligent driving decisions and thus improving driving safety and driving experience.

[0006] In some embodiments, the road surface identification results include at least one of the following: road surface type and road surface features.

[0007] In some embodiments, the above-mentioned road surface characteristics include at least one of the following: softness, slope, rolling resistance, smoothness, adhesion coefficient, and slip ratio.

[0008] In some embodiments, the above-mentioned road surface types include at least one of the following: snow, mud, grassland, sand, mountain, rock, wading, and asphalt.

[0009] In some embodiments, the controller is configured to determine the road surface identification result of the road surface on which the vehicle is traveling based on sensor data sent by multiple sensors, including: determining the softness of the road surface on which the vehicle is traveling based on sensor data sent by a tire noise sensor.

[0010] In some embodiments, the above-described plurality of sensors further include at least one of a vision sensor and a tactile sensor.

[0011] In some embodiments, the tactile sensor includes at least one of the following: an inertial navigation sensor, a suspension height sensor, a wheel speed sensor, a wheel rotation speed sensor, a torque sensor, a steering wheel angle sensor, a brake pedal sensor, and a brake torque sensor.

[0012] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors, including: determining the initial road surface recognition result corresponding to each sensor based on the sensor data sent by each sensor; and determining the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor.

[0013] In some embodiments, the initial road surface identification result includes the original probability corresponding to each candidate road surface type; the road surface identification result includes the road surface type of the road on which the vehicle travels.

[0014] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor, including: if the road surface type in the initial road surface recognition result is consistent, using the initial road surface recognition result as the road surface recognition result of the road where the vehicle is traveling.

[0015] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor, and further includes: in the case that the road surface types in the initial road surface recognition results are inconsistent, determining the probability difference between the initial road surface recognition results corresponding to each sensor based on the original probability of the initial road surface recognition results corresponding to each sensor; and if the probability difference meets a preset condition, taking the initial road surface recognition result corresponding to the maximum original probability as the road surface recognition result of the road where the vehicle is traveling.

[0016] In some embodiments, the preset conditions include: the probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest; and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

[0017] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor, and further includes: determining the comprehensive probability of each candidate road surface type based on the original probability corresponding to each candidate road surface type in the initial road surface recognition result corresponding to each sensor; and determining the road surface type of the road where the vehicle is traveling based on the comprehensive probability of each candidate road surface type.

[0018] In some embodiments, the controller is configured to determine the comprehensive probability of each candidate road surface type based on the original probabilities of each candidate road surface type in the initial road surface recognition results corresponding to each sensor, including: weighting and fusing the original probabilities of the same candidate road surface type in the initial road surface recognition results corresponding to each sensor to obtain the comprehensive probability of the candidate road surface type.

[0019] In some embodiments, the weights corresponding to the visual sensors in the weighted fusion process are determined based on the visibility of the vehicle's environment.

[0020] In some embodiments, during weighted fusion, the weights corresponding to the tire noise sensors are determined based on at least one of the noise of the vehicle's environment and the vehicle's motion state.

[0021] In some embodiments, the weights of the tactile sensors are determined based on the vehicle's motion state during weighted fusion.

[0022] In some embodiments, during the weighted fusion process, the weights corresponding to each sensor are determined as follows: based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined; based on the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor, the weights corresponding to each sensor are determined.

[0023] In some embodiments, the controller is configured to determine the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road on which the vehicle travels, including: determining the attention score of the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on an attention mechanism; and determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the attention score of each sensor.

[0024] In some embodiments, the controller is configured to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention score of each sensor, including: normalizing the attention score of each sensor to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

[0025] In some embodiments, the controller is further configured to: if the road surface types in the initial road surface recognition results are consistent, before using the initial road surface recognition results as the road surface recognition results of the road where the vehicle is traveling, filter the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor.

[0026] In some embodiments, the controller is configured to filter the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor, including: deleting the initial road surface recognition results containing snow when the ambient temperature is greater than a preset temperature threshold, to obtain the filtered initial road surface recognition results corresponding to each sensor.

[0027] In some embodiments, the controller is configured to determine the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type, including: smoothing the comprehensive probability of each candidate road surface type using a moving average function to determine the smoothed comprehensive probability of each candidate road surface type; and determining the road surface type of the vehicle's driving road based on the smoothed comprehensive probability of each candidate road surface type.

[0028] In some embodiments, the controller is configured to determine the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type, including: determining the candidate road surface type corresponding to the maximum comprehensive probability as the road surface type of the vehicle's driving road.

[0029] In some embodiments, the tire noise sensor described above includes tire noise sensors for each wheel in at least one wheel.

[0030] In some embodiments, the controller is configured to determine the road surface identification result of the road where the vehicle is traveling based on sensor data sent by multiple sensors, including: determining the tire noise characteristics of each wheel based on sensor data sent by the tire noise sensors of each wheel; and determining the road surface identification result of the road where the vehicle is traveling based on the tire noise characteristics of each wheel.

[0031] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the tire noise characteristics of each wheel, including: determining the road surface recognition result of each wheel based on the tire noise characteristics of each wheel; and determining the road surface recognition result of the road where the vehicle is traveling based on the road surface recognition result of each wheel.

[0032] In some embodiments, determining the road surface recognition result of each wheel based on the tire noise characteristics of each wheel includes: fusing the tire noise characteristics of each wheel to obtain the fused tire noise characteristics of each wheel; and determining the road surface recognition result of each wheel based on the fused tire noise characteristics of each wheel.

[0033] In some embodiments, the above-mentioned fusion of the tire noise features of each wheel to obtain the fused tire noise features of each wheel includes: feature splicing of the tire noise features of each wheel to obtain the fused tire noise features of each wheel.

[0034] In some embodiments, the above-mentioned tire noise characteristics include at least one of the following: statistical characteristics, Mel time-frequency characteristics, and wavelet transform time-frequency characteristics.

[0035] In some embodiments, the controller is further configured to: determine whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode when the current driving mode and the target driving mode are inconsistent.

[0036] In some embodiments, the controller is specifically configured to: determine that the vehicle will switch from the current driving mode to the target driving mode when the safety level of the current driving mode is lower than that of the target driving mode; or, determine that the vehicle will switch from the current driving mode to the target driving mode when the safety level of the current driving mode is higher than that of the target driving mode and the vehicle meets the driving mode switching conditions.

[0037] In some embodiments, the controller is further configured to determine a target driving mode based on the road surface recognition results of the road where the vehicle is traveling.

[0038] In some embodiments, the vehicle control system further includes an actuator connected to the controller; the controller is further configured to control the actuator to perform adjustment actions corresponding to the target driving mode.

[0039] In some embodiments, the actuators described above include at least one of the following: an electronic differential, a power subsystem, a braking subsystem, a steering subsystem, and an active suspension subsystem.

[0040] In some embodiments, the controller includes an intelligent driving domain controller; the intelligent driving domain controller is configured to: receive sensor data sent by sensors; and determine the road surface recognition result of the road where the vehicle is traveling based on the sensor data sent by multiple sensors.

[0041] In some embodiments, the controller further includes a vehicle controller; the intelligent driving domain controller is further configured to send road surface recognition results of the road where the vehicle is traveling to the vehicle controller.

[0042] In some embodiments, the controller further includes a vehicle controller; the intelligent driving domain controller is further configured to: determine a target driving mode based on the road surface recognition results of the road where the vehicle is traveling; send the target driving mode to the vehicle controller; and the vehicle controller is configured to: control the actuator to perform the adjustment action corresponding to the target driving mode.

[0043] In some embodiments, the vehicle control system further includes: a human-machine interface device connected to the controller; the human-machine interface device is configured to: generate a road surface recognition command and send the road surface recognition command to the controller; the controller is configured to: in response to the road surface recognition command, determine the road surface recognition result of the road surface on which the vehicle is traveling based on sensor data sent by multiple sensors.

[0044] In some embodiments, the road surface recognition command is triggered by at least one of the following methods: receiving a touch operation from a user on the display interface of an in-vehicle display device; receiving a trigger operation from a user on the road surface recognition button; or receiving a sound signal from a user carrying out road surface recognition.

[0045] In some embodiments, the controller is further configured to send road surface recognition results to a human-machine interaction device; the human-machine interaction device is further configured to receive and display the road surface recognition results sent by the controller.

[0046] In some embodiments, the controller is further configured to: send an information display instruction carrying prompt information to the human-machine interaction device when the current driving mode and the target driving mode of the vehicle are inconsistent; the prompt information includes: road recognition results and the target driving mode; the human-machine interaction device is further configured to: receive and parse the information display instruction to obtain the prompt information; and display the prompt information.

[0047] In some embodiments, the above-mentioned prompt information is displayed in at least one of the following ways: displaying the prompt information through the display interface of the in-vehicle display device; or outputting the prompt information through the in-vehicle sound device.

[0048] In some embodiments, the human-machine interface device is further configured to: generate a confirmation command for a target driving mode; and the controller is further configured to: control the vehicle to drive based on the target driving mode in response to the confirmation command for the target driving mode.

[0049] In some embodiments, the confirmation command for the target driving mode is triggered by at least one of the following methods: receiving a driving mode confirmation operation from a user on the display interface of the in-vehicle display device; receiving a user's trigger operation on the driving mode confirmation button; or receiving a sound signal from the user confirming the target driving mode.

[0050] In some embodiments, the controller is configured to determine the road surface recognition result of the road being traveled by the vehicle based on sensor data sent by multiple sensors, including: segmenting the sensor data sent by the vision sensor to obtain multiple regional sub-images; the sensor data includes: road surface images of the road currently being traveled by the vehicle; filtering the categories of the multiple regional sub-images to determine regional sub-images of the category of road surface; the category of the regional sub-images includes at least one of the following: road surface, obstacle, lane line; and determining the road surface recognition result of the road currently being traveled by the vehicle based on the regional sub-images of the category of road surface.

[0051] In some embodiments, the controller is configured to segment a road surface image of the road currently being traveled by the vehicle to obtain multiple regional sub-images, including: performing semantic segmentation of the road surface image based on a neural network model to determine the category of each pixel in the road surface image; and dividing the road surface image into multiple regional sub-images based on the category of each pixel.

[0052] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors, including: extracting features from sensor data sent by tactile sensors to obtain the motion features of the vehicle; determining the road surface features of the road where the vehicle is currently traveling based on the motion features; and determining the road surface recognition result of the road where the vehicle is currently traveling based on the road surface features.

[0053] In some embodiments, the above-mentioned motion characteristics include at least one of the following: four-wheel load, vehicle speed, gradient, rolling resistance, slip ratio, coefficient of adhesion, and road surface smoothness.

[0054] Secondly, a vehicle control method is also provided, including: acquiring sensor data from multiple sensors; wherein the multiple sensors include at least a tire noise sensor; and determining the road surface identification result of the road surface on which the vehicle is traveling based on the sensor data sent by the multiple sensors.

[0055] In some embodiments, determining the road surface identification result of the road where the vehicle is traveling based on multiple sensor data includes: if the road surface type in the initial road surface identification result is consistent, using the initial road surface identification result as the road surface identification result of the road where the vehicle is traveling.

[0056] In some embodiments, the above-mentioned determination of the road surface recognition result of the road where the vehicle is traveling based on multiple sensor data further includes: when the road surface types in the initial road surface recognition results are inconsistent, determining the probability difference between the initial road surface recognition results corresponding to each sensor based on the original probability of the initial road surface recognition results corresponding to each sensor; and when the probability difference meets a preset condition, taking the initial road surface recognition result corresponding to the maximum confidence level as the road surface recognition result of the road where the vehicle is traveling.

[0057] In some embodiments, the preset conditions include: the probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest; and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

[0058] In some embodiments, the above-mentioned determination of the road surface recognition result of the vehicle's driving road based on the initial road surface recognition result corresponding to each sensor further includes: determining the comprehensive probability of each candidate road surface type based on the original probability corresponding to each candidate road surface type in the initial road surface recognition result corresponding to each sensor; and determining the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type.

[0059] In some embodiments, determining the comprehensive probability of each candidate road type based on the original probabilities of each candidate road type in the initial road recognition results corresponding to each sensor includes: weighting and fusing the original probabilities of the same candidate road type in the initial road recognition results corresponding to each sensor to obtain the comprehensive probability of the candidate road type.

[0060] In some embodiments, during the weighted fusion process, the weights corresponding to each sensor are determined as follows: based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined; based on the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor, the weights corresponding to each sensor are determined.

[0061] In some embodiments, determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road on which the vehicle travels includes: determining the attention score between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on an attention mechanism; and determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the attention score of each sensor.

[0062] In some embodiments, determining the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention scores of each sensor includes: normalizing the attention scores of each sensor to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

[0063] Thirdly, an electronic device is also provided, comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional vehicle control methods of the second aspect described above.

[0064] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed by a device, enable the device to perform any of the optional vehicle control methods described in the second aspect above.

[0065] Fifthly, this application provides a vehicle, including: the vehicle control system of the first aspect, or the electronic equipment of the third aspect, or the computer-readable storage medium of the fourth aspect.

[0066] In a sixth aspect, this application provides a computer program product including computer instructions that, when executed on a processor of a device, enable the device to perform any of the optional vehicle control methods described in the second aspect above. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the structure of a vehicle control system provided in an embodiment of this application;

[0069] Figure 2 This is a schematic diagram showing the location of a tire noise sensor provided in an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of another vehicle control system provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram of the architecture of a dynamic weight update method for an attention learning mechanism provided in an embodiment of this application;

[0072] Figure 5A schematic diagram of a process for determining road surface recognition results provided in an embodiment of this application;

[0073] Figure 6 A schematic diagram illustrating the process of determining and switching a target driving mode, as provided in an embodiment of this application;

[0074] Figure 7 This application provides a schematic diagram of control signal interaction in a vehicle control system.

[0075] Figure 8 A schematic flowchart of a vehicle control method provided in an embodiment of this application;

[0076] Figure 9 This is a schematic diagram of the structure of a vehicle control device provided in an embodiment of this application;

[0077] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0078] Figure label:

[0079] Controller 101; Sensor 102; Tire noise sensor 1021; Vision sensor 1022; Tactile sensor 1023; Actuator 103; Intelligent driving domain controller 1011; Vehicle controller 1012; Human-machine interaction device 104. Detailed Implementation

[0080] In the embodiments of this application, the terms "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," "fourth," "fifth," and "sixth" may explicitly or implicitly include one or more of that feature.

[0081] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0082] As the need for vehicles to navigate complex scenarios increases, these scenarios place higher demands on vehicle performance. For example, when driving on uneven roads, vehicles are affected by the bumps and undulations, impacting both vehicle adaptability and user comfort. To address this issue, identifying the road surface type and matching it with an appropriate driving mode can improve vehicle adaptability and driving experience in complex scenarios. Therefore, achieving road surface recognition is a pressing problem that needs to be solved.

[0083] Based on this, this application proposes a vehicle control system, method, electronic device, storage medium, and vehicle, including: a controller and multiple sensors connected to the controller, the multiple sensors including at least: a tire noise sensor; the controller is configured to: receive sensor data sent by the multiple sensors; and determine the road surface identification result of the road surface on which the vehicle is traveling based on the sensor data sent by the multiple sensors. The vehicle control system of this application can collect sensor data during vehicle travel at least through the tire noise sensor, and the controller can identify the road surface on which the vehicle is traveling based at least on the sensor data from the tire noise sensor, thereby providing a basis for intelligent driving decisions and improving driving safety and driving experience.

[0084] The vehicle control system, method, electronic equipment, storage medium, and vehicle of this application will be described below with reference to the accompanying drawings.

[0085] It should be noted that the vehicle control system provided in this application embodiment can be installed in a vehicle, which can be a fuel vehicle, a pure electric vehicle, or a hybrid vehicle, and there is no limitation here.

[0086] like Figure 1 As shown, the vehicle control system of this application includes a controller 101 and various sensors 102 connected to the controller. The various sensors 102 include at least a tire noise sensor 1021.

[0087] As an information acquisition unit, the sensor collects relevant information for road surface identification. Specifically, the tire noise sensor collects tire noise signals in real time while the vehicle is in motion.

[0088] The controller, as a processing unit, is responsible for receiving sensor data and performing road surface recognition calculations.

[0089] Among them, the tire noise sensor 1021 can be a microphone, which can acquire tire noise signals by collecting the noise signals generated by the friction between the tire and the road surface, thereby providing data support for road surface recognition.

[0090] For example, such as Figure 2As shown, the microphone can be installed at positions 1-5. Position 1 indicates installation inside the wheel hub cover, which reduces external environmental noise interference and provides some dust and water protection. Positions 2, 3, and 4 indicate installation on the wheel arch, where external noise interference is minimal and the fixed position facilitates wired transmission of tire noise signals to the controller. Position 5 indicates installation inside the wheel, where external noise interference is minimal, the fixed position provides high dust and water protection, and tire noise signals are transmitted to the controller wirelessly.

[0091] Continue reading Figure 1 The multiple sensors 102 also include at least one of a vision sensor 1022 and a tactile sensor 1023.

[0092] Understandably, when identifying road surfaces, in addition to auditory perception, tactile perception and visual perception can also be used. Accordingly, the vehicle control system needs to add a visual sensor 1022 and a tactile sensor 1023 to provide data support for road surface identification.

[0093] The vision sensor 1022 is typically a camera. The sensor data of the vision sensor is usually a road image. The camera can be a vehicle-mounted camera, a camera installed on both sides of the road that can communicate with the vehicle, or a camera mounted on other vehicles or equipment. The camera can be a forward-looking wide-angle camera, a surround-view camera, or an in-vehicle monitoring camera; this application does not impose specific limitations in this regard.

[0094] The vision sensor 1022 can also be used with devices such as lidar, millimeter-wave radar, and infrared cameras to help identify road surface materials, water accumulation, ice formation, and other conditions.

[0095] like Figure 3 As shown, the tactile sensor 1023 includes at least one of the following: an inertial navigation sensor, a suspension height sensor, a wheel speed sensor, a wheel speed sensor, a torque sensor, a steering wheel angle sensor, a brake pedal sensor, and a brake torque sensor.

[0096] Among them, the inertial navigation sensor is used to acquire the acceleration and angular velocity information of the vehicle along three directions (x, y, z). The change in acceleration in the vertical direction can reflect the change in road surface smoothness, the change in longitudinal acceleration can reflect the change in road surface slope, and the lateral acceleration and angular velocity can reflect the vehicle's roll, and thus reflect the change in road surface slope. Based on the acceleration and angular velocity information of the vehicle along the three directions (x, y, z), road surface characteristics such as smoothness and slope can be obtained.

[0097] The suspension height sensor provides information on the travel of the four-wheel suspension. When there are potholes or bumps on the road surface, the suspension will undergo significant compression or extension due to the impact. The change in suspension travel can reflect the unevenness of the road surface and thus determine the smoothness of the road surface.

[0098] The linear velocity of the wheel relative to the ground, provided by the wheel speed sensor, can reflect the adhesion characteristics and slip state between the road surface and the tire, thereby determining the road surface's adhesion coefficient, slip ratio, and other characteristics.

[0099] The information such as the speed and torque of each wheel provided by the wheel speed and torque sensors can reflect the rolling resistance and load characteristics of the road surface on the wheels, thereby determining the softness, rolling resistance and slope of the road surface.

[0100] The steering wheel angle sensor provides steering wheel angle and angular velocity information, which can reflect differences in lateral grip and road surface roughness, and thus determine road surface characteristics such as adhesion coefficient and slip ratio.

[0101] The brake pedal sensor and brake torque sensor provide information such as brake activation signals and brake torque values ​​to determine the road surface adhesion coefficient and slip ratio.

[0102] Therefore, sensor data is acquired through sensors such as inertial navigation sensors, suspension height sensors, wheel speed sensors, wheel speed sensors, torque sensors, steering systems, and braking systems, providing data support for road surface recognition through tactile perception.

[0103] Continue reading Figure 1 The vehicle control system also includes an actuator 103 connected to the controller 101.

[0104] Actuator 103 is used to execute the target driving mode determined by controller 101 based on road surface recognition results.

[0105] The actuator 103 includes at least one of the following: electronic differential, power subsystem, braking subsystem, steering subsystem, and active suspension subsystem.

[0106] It should be noted that, in addition to execution, the powertrain subsystem, braking subsystem, steering subsystem, and active suspension subsystem also have data acquisition capabilities. Therefore, the functions of the suspension height sensor, steering wheel angle sensor, brake pedal sensor, and brake torque sensor in the tactile sensors can be realized by the powertrain subsystem, braking subsystem, and active suspension subsystem.

[0107] Continue reading Figure 1 The controller 101 includes a smart driving domain controller 1011 and a vehicle controller 1012.

[0108] The intelligent driving domain controller 1011 is used to identify the road surface and determine the target driving mode based on sensor data sent by multiple sensors, and the vehicle controller 1012 is used to control the actuator 103 to perform the adjustment actions corresponding to the target driving mode.

[0109] Continue reading Figure 1 The vehicle control system also includes a human-machine interface device 104 connected to the controller 101.

[0110] The human-machine interaction device 104 includes at least: a multimedia host, a central control panel, a driving mode selection button, and a combination instrument.

[0111] The process of road surface recognition by the vehicle control system of this application is described below.

[0112] In some embodiments, the controller is configured to receive sensor data from multiple sensors and, based on the sensor data from multiple sensors, determine the road surface recognition result of the road where the vehicle is traveling.

[0113] As one possible implementation, the controller analyzes the sensor data according to a preset algorithm and determines the corresponding road surface recognition result based on the correlation between the sensor data and the road surface recognition result. The correlation between the sensor data and the road surface recognition result can be determined using a deep learning model.

[0114] In one possible implementation, the correlation between sensor data and road surface recognition results can be defined as the correlation between each type of sensor data and road surface recognition results. The road surface recognition results corresponding to each type of sensor are determined, and the road surface recognition results of the road where the vehicle is traveling are obtained by fusing the road surface recognition results corresponding to each type of sensor.

[0115] In another possible implementation, the relationship between sensor data and road surface recognition results can be the relationship between fused sensor data and road surface recognition results. First, various types of sensor data are fused to obtain fused sensor data. Then, by learning the relationship between sensor data and road surface recognition results, the road surface recognition result of the road where the vehicle is traveling can be determined.

[0116] The road surface recognition results include at least one of the following: road surface type and road surface features.

[0117] Understandably, road surface type is used to define the road surface for vehicle driving in a specific context, belonging to the road surface classification process, while road surface feature representation is a refined description of the physical properties of the road surface, usually involving specific numerical values ​​or relative magnitude relationships. Road surface type and road surface features provide a basis for precise vehicle control and driving decisions, thereby ensuring both vehicle safety and comfort.

[0118] Road surface characteristics include at least one of the following: softness, slope, rolling resistance, smoothness, adhesion coefficient, and slip ratio.

[0119] The road surface type includes at least one of the following: snow, mud, grassland, sand, mountain, rock, water crossing, and asphalt.

[0120] Due to differences in texture, hardness, and roughness of different road surfaces, the frequency and amplitude of tire noise signals will vary. By analyzing the tire noise signals collected by the tire noise sensor, the controller can determine the current road surface identification result and thus complete the road surface identification.

[0121] In some embodiments, since the softness of the road surface can change the contact mode and friction characteristics between the wheel and the ground, thereby causing changes in the frequency, amplitude, etc. of the tire noise signal, the softness of the road surface can be analyzed based on the tire noise signal. The controller is configured to determine the softness of the road surface on which the vehicle is traveling based on the sensor data sent by the tire noise sensor.

[0122] As one possible implementation, the tire noise sensor collects sensor data, i.e. tire noise signal, in real time while the vehicle is driving. The time-domain tire noise signal is converted into quantifiable features through signal processing algorithms, and a mapping relationship between the features and the softness level is established. Based on the mapping relationship between the features and the softness level, the softness of the road on which the vehicle is driving is obtained.

[0123] Among them, the higher the softness, the larger the proportion of the low frequency range. The quantifiable feature is the proportion of the low frequency range. The time-domain tire noise signal is converted into the frequency-domain tire noise signal to obtain the respective proportions of low, medium and high frequency tire noise signals.

[0124] The softness can be divided into different intervals to obtain different softness grades, such as hard, medium, soft, etc.

[0125] As another possible approach, a softness determination model can be established and trained using historical tire noise signals and their corresponding softness values. The trained softness determination model can then be obtained by inputting real-time detected tire noise signals into it.

[0126] Therefore, by using the sensor data from the tire noise sensor to calculate the softness of the road surface the vehicle is traveling on, the system provides the vehicle with microscopic road surface characteristic data, thereby helping intelligent decision-making. For example, when the mud is soft, the system can automatically reduce the tire pressure to prevent the vehicle from getting stuck, thus improving passability and safety in complex road conditions.

[0127] In some embodiments, the tire noise sensor includes tire noise sensors for each wheel in at least one wheel.

[0128] It should be noted that different wheels may produce different tire noise signals due to differences in force and contact with the road surface. In order to comprehensively capture the tire noise signals of different wheels, each wheel is equipped with at least one tire noise sensor. When performing road surface recognition, the tire noise signals of multiple or even all wheels are usually detected to ensure the accuracy of the road surface recognition results.

[0129] The installation location of the tire noise sensor can be referred to the description in the above embodiments, and will not be repeated here.

[0130] In some embodiments, the controller is configured to: determine the tire noise characteristics of each wheel based on sensor data sent by the tire noise sensors of each wheel; and determine the road surface identification result of the road on which the vehicle is traveling based on the tire noise characteristics of each wheel.

[0131] In some embodiments, the tire noise characteristics of the wheels include at least one of the following: statistical characteristics, Mel-time frequency characteristics, and wavelet transform time frequency characteristics. The above-mentioned determination of the road surface recognition result based on the tire noise characteristics of each wheel can be achieved by: fusing the tire noise characteristics of each wheel to obtain fused tire noise characteristics of each wheel; and determining the road surface recognition result of each wheel based on the fused tire noise characteristics of each wheel.

[0132] Among them, statistical features represent deep learning features extracted from descriptive statistics such as mean, variance, standard deviation, median, quartiles, maximum, minimum, and root mean square of the tire noise signal; Mel time-frequency features represent deep learning features extracted from the Mel time-frequency plot of the tire noise signal; and wavelet transform time-frequency features represent deep learning features extracted from the wavelet transform time-frequency plot of the tire noise signal.

[0133] In some embodiments, the above-mentioned determination of the road surface recognition result of each wheel based on the fused tire noise features of each wheel can be implemented as follows: performing feature stitching on the tire noise features of each wheel to obtain the fused tire noise features of each wheel.

[0134] For example, the road surface recognition result for each wheel can be determined based on a multi-feature fusion tire noise recognition model. The multi-feature fusion tire noise recognition model includes: a deep learning feature extraction layer, a feature fusion layer, and a classifier. The deep learning feature extraction layer extracts deep learning features from descriptive statistics, Mel time-frequency plots, and wavelet transform time-frequency plots. The feature fusion layer fuses the deep learning features from the statistical features, Mel time-frequency plots, and wavelet transform time-frequency plots through feature concatenation. The classifier identifies the fused features.

[0135] Obtaining the Mel-frequency map first requires performing a short-time Fourier transform on the tire noise signal to obtain its spectrum. In this application, the microphone sampling frequency is 44.1 kHz. Before performing the short-time Fourier transform, the tire noise signal needs to be framed and windowed. The frame length is 2205, and the overlap between frames is 1764. A Hamming window with a window length of 2205 is used for windowing. Then, a Fourier transform is performed on each frame with a Fourier length of 4096. Next, the spectrum is filtered using a Mel filter bank (64 Mel filters). After processing, logarithmic operations are performed to finally obtain the Mel-frequency map of the tire noise signal, which serves as the input to the multi-feature fusion tire noise recognition model.

[0136] The wavelet transform time-frequency graph is generated by performing a wavelet transform on the tire noise signal, with the wavelet basis function being the Morse function. This wavelet transform time-frequency graph is then converted into a 224*224 RGB image, which serves as the input to the multi-feature fusion tire noise recognition model.

[0137] The deep learning feature extraction layer of the feature fusion tire noise recognition model targets three types of input data, each corresponding to a different deep learning feature extraction network: a Multi-Layer Perceptron (MLP), a Deep Convolutional Neural Network (DCNN), and a Deep Residual Network (DRN). The MLP extracts statistical features, the DCNN extracts Mel-time-frequency features, and the DRN extracts wavelet transform-time-frequency features.

[0138] The MLP consists of one input layer, three hidden layers, and one output layer. The size of the input layer is equal to the number of descriptive statistics, the sizes of the three hidden layers are 64, 256, and 512, respectively, and the size of the output layer is equal to the number of deep learning features extracted, with a value of 96. Each layer in the MLP is implemented through a fully connected layer, and a ReLU activation function is added after each fully connected layer to enhance the network's non-linear fitting ability.

[0139] A deep convolutional neural network (CNN) is an 83-layer neural network model containing 13 identical convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer, an activation function layer, a group convolutional layer, another batch normalization layer, and another activation function layer. The kernel size of the convolutional layers is 3x3. Group convolutions are used to reduce model parameters; the kernel size is 3x3, and the number of convolutional groups is equal to the number of channels in the input feature map. Following the last convolutional module are a convolutional layer, a batch normalization layer, an activation function layer, a global average pooling layer, and a fully connected layer. The output of the fully connected layer is the number of features extracted by deep learning, with a value of 96.

[0140] The Deep Residual Network (DRN) is a deep residual network with an attention mechanism, comprising eight residual modules with this mechanism. The wavelet transform time-frequency map serves as input, first passing through a 7x7 convolutional kernel with a stride of 2, then sequentially through a batch normalization layer, an activation function layer, and a 3x3 max-pooling layer with a stride of 2 before being fed into the attention-based residual modules for feature extraction. These modules consist of a main path and a shortcut path. The main path includes convolutional and attention modules. The convolutional module comprises convolutional layers, a batch normalization layer, an activation function layer, another convolutional layer, and another batch normalization layer for feature extraction. The attention module consists of a global average pooling layer, a fully connected layer, an activation function layer, another fully connected layer, and another activation function layer. Its final activation function layer is a sigmoid function, used to fix the output value between 0 and 1 as weights. Finally, the feature map extracted by the convolutional module is multiplied by the weight information extracted by the attention module, resulting in the output of the main path. The Shortcut path contains convolutional layers and batch normalization layers for feature extraction. After feature extraction from the main path and the Shortcut path, the two feature maps are summed, and then passed through an activation function layer to serve as the output of the residual module with attention mechanism. Following the last residual module with attention mechanism is a global average pooling layer and a fully connected layer. The output of the fully connected layer is the number of features extracted by deep learning, with a value of 96.

[0141] The feature fusion layer concatenates three 96-dimensional feature vectors into a single 288-dimensional feature vector through feature map concatenation.

[0142] The classifier consists of a fully connected layer, an activation function layer, a dropout layer, and a softmax layer. The first fully connected layer has an input size of 2^88 and an output size of 5^12; the activation function layer uses ReLU; the dropout layer has a dropout rate of 0.2 to increase the model's robustness; the last fully connected layer has an input size of 5^12 and an output size equal to the number of terrain categories; finally, the softmax layer obtains the predicted probability (initial probability), i.e., the confidence score, for each terrain category.

[0143] In some embodiments, the above-mentioned determination of the road surface recognition result of the vehicle's travel path based on the tire noise characteristics of each wheel can be implemented as follows: determining the road surface recognition result of each wheel based on the tire noise characteristics of each wheel; and determining the road surface recognition result of the vehicle's travel path based on the road surface recognition result of each wheel.

[0144] It is understandable that when a vehicle is in motion, each wheel is usually in the same road environment. However, there are also some special road environments where the road surface type of each wheel is different. In this case, it is necessary to make further judgments on the road surface recognition results of the vehicle's driving road based on the road surface recognition results of each wheel.

[0145] In one possible implementation, if the road surface recognition results of multiple or all wheels are the same, then the road surface recognition results of multiple wheels can be used as the road surface recognition results of the road on which the vehicle is traveling.

[0146] In another possible implementation, if the road surface recognition results for the left and right wheels consistently differ, then the road surface recognition result that has a greater impact on driving can be used as the road surface recognition result for the vehicle's driving path. For example, if the road surface types for the left and right wheels are snow and asphalt, since snow has a greater impact on vehicle form, snow can be used as the road surface type for the current vehicle's driving path.

[0147] Furthermore, if the road surface recognition results of the left and right wheels continue to differ, the road surface recognition results of the road the vehicle is traveling on can be divided into the road surface recognition results of the left and right sides, and the road surface recognition results of the left and right wheels can be used as the road surface recognition results of the road the vehicle is traveling on, thereby achieving differentiated control of the left and right wheels.

[0148] It should be noted that before determining the tire noise characteristics of each wheel, the tire noise signal can be low-pass filtered to effectively filter out high-frequency environmental noise mixed in the tire noise signal. At the same time, the tire noise signal can be enhanced to make the subsequently extracted tire noise features more accurately reflect the friction interaction characteristics between the tire and the road surface, providing a data foundation for road surface type identification and road surface feature judgment based on tire noise.

[0149] In some embodiments, the process of identifying the road surface based on sensor data sent by a vision sensor can be as follows:

[0150] The controller is configured to: segment sensor data sent by the vision sensor to obtain multiple regional sub-images; the sensor data includes: road surface image of the road currently being driven by the vehicle; filter the categories of the multiple regional sub-images to determine the regional sub-images of the category of road surface; the category of the regional sub-images includes at least one of the following: road surface, obstacle, lane line; and determine the road surface recognition result of the road currently being driven by the vehicle based on the regional sub-images of the category of road surface.

[0151] It should be noted that, in addition to obtaining images of the road surface where the vehicle is currently traveling, the road surface image can also be obtained by combining the vehicle's positioning information with satellite remote sensing images of the vehicle's current location, which can serve as a supplement to the road surface image obtained by the visual sensor.

[0152] In some embodiments, the above-mentioned segmentation of sensor data sent by the vision sensor to obtain multiple regional sub-images can be implemented as follows: performing semantic segmentation of the road surface image based on a neural network model to determine the category of each pixel in the road surface image; and dividing the road surface image into multiple regional sub-images based on the category of each pixel.

[0153] For example, a neural network is used to perform semantic segmentation on the image, analyzing the image pixel by pixel and assigning a category label (road surface, lane lines, obstacle) to each pixel. The image is divided into different regions according to the category label. All pixels with the category label "road surface" are extracted from the segmented image, and pixels of other categories are filtered out to obtain multiple sub-images of regions with the category label "road surface".

[0154] In some embodiments, the above-mentioned determination of the road surface recognition result of the vehicle's current driving road based on the regional sub-images of the road surface category can be implemented as follows: inputting the above-mentioned multiple regional sub-images into a classification neural network, the classification neural network extracts features from the regional sub-images to obtain features such as road surface texture and road surface color, classifies the road surface based on the road surface texture, road surface color and other features, and generates a road surface classification result, that is, the road surface type corresponding to the road surface image and the corresponding confidence level (initial probability).

[0155] It should be noted that when the controller and vision sensor communicate, the communication process is not real-time due to communication transmission delay. Therefore, the vision sensor usually acquires road images of the road the vehicle is traveling on, generates a continuous video stream, and sends the video stream to the controller. The controller extracts single-frame images from the video stream for processing, which allows the controller to acquire a complete sequence of road images even under non-real-time conditions.

[0156] In some embodiments, the process of identifying the road surface based on sensor data sent by the tactile sensor can be as follows:

[0157] The controller is configured to: extract features from sensor data sent by the tactile sensor to obtain the vehicle's motion features; determine the road surface features of the road surface the vehicle is currently traveling on based on the motion features; and determine the road surface recognition result of the road surface the vehicle is currently traveling on based on the road surface features.

[0158] The motion characteristics include at least one of the following: four-wheel load, vehicle speed, gradient, rolling resistance, slip ratio, coefficient of adhesion, and road surface smoothness. The sensor data transmitted by the tactile sensor includes at least one of the following: acceleration, steering angle, vehicle speed, and suspension displacement.

[0159] In some embodiments, the above-mentioned feature extraction of sensor data sent by the tactile sensor to obtain the vehicle's motion characteristics can be implemented as follows: feature extraction of sensor data sent by the tactile sensor based on the vehicle dynamics model to obtain the vehicle's motion characteristics.

[0160] In some embodiments, the above-mentioned determination of the road surface features of the road currently being traveled by the vehicle based on the vehicle's motion characteristics, and determination of the road surface recognition result of the road currently being traveled by the vehicle based on the road surface features, can be implemented as follows: based on the mapping relationship between motion features and road surface features, determine the road surface features corresponding to the motion features, compare the road surface features with the feature benchmark of the road surface type, and determine the road surface type of the road currently being traveled by the vehicle.

[0161] The mapping relationship between motion features and road surface features can be direct or indirect.

[0162] In some embodiments, the above-mentioned comparison of road surface features with the feature reference of road surface type to determine the road surface type of the road currently being driven by the vehicle can be implemented as follows: determining the numerical range of road surface features such as slope, rolling resistance, and adhesion coefficient for each road surface type to obtain the feature reference of the road surface type, and matching the current road surface features of the vehicle with the feature reference of the road surface type to obtain the road surface type.

[0163] In other embodiments, the process of identifying the road surface based on sensor data sent by the tactile sensor can be implemented as follows: training a deep learning model based on sensor data sent by historical tactile sensors to obtain a trained model, inputting the sensor data of the vehicle's real-time tactile sensor into the trained deep learning model to obtain the road surface identification result of the road currently being driven by the vehicle.

[0164] It is understandable that changes in vehicle motion characteristics are caused by changes in road surface characteristics. By analyzing the patterns of these changes, the road surface characteristics can be accurately determined. Road surface characteristics are physical attributes of the road, and road surface types are classifications based on these characteristics. By establishing the correlation between road surface characteristics and road surface types, the road surface type can be accurately identified, thus yielding the road surface identification result for the road the vehicle is currently traveling on.

[0165] It should be noted that, to ensure the accuracy of road surface recognition results, after the controller receives the raw data sent by the tactile sensor, it can uniformly convert signals from different units to ensure data format consistency, providing a unified benchmark for subsequent feature extraction and fusion analysis. Filtering algorithms can also be used to remove noise signals from the sensor data, ensuring data smoothness and reliability. Furthermore, signal limiting processing can be applied to ensure that the sensor data remains within a reasonable physical range.

[0166] In some embodiments, where the road surface recognition result includes the road surface type of the road where the vehicle is traveling, and there are multiple types of sensors, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors, including: determining the initial road surface recognition result corresponding to each sensor based on the sensor data sent by each sensor; and determining the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor.

[0167] Therefore, the road surface recognition results of a single sensor are easily limited, while by fusing the initial road surface recognition results of multiple sensors, the recognition error of a single sensor can be offset, thereby improving the accuracy of the road surface recognition results.

[0168] In some embodiments, the initial road surface identification result includes the original probabilities corresponding to the candidate road surface types. Determining the road surface identification result of the road the vehicle is traveling on based on the initial road surface identification results corresponding to each sensor can be achieved by: determining the comprehensive probability of each candidate road surface type based on the original probabilities corresponding to each candidate road surface type in the initial road surface identification results corresponding to each sensor; and determining the road surface type of the road the vehicle is traveling on based on the comprehensive probability of each candidate road surface type.

[0169] In some embodiments, the controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor. This can be implemented as follows: if the road surface type in the initial road surface recognition result is consistent, the initial road surface recognition result is used as the road surface recognition result of the road where the vehicle is traveling.

[0170] In this case, when the controller is configured to determine the straight line of the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor, it can first be determined whether the road surface type in the initial road surface recognition result corresponding to each sensor is consistent.

[0171] In some embodiments, when the road surface types in the initial road surface recognition results are inconsistent, the controller is configured to: determine the probability difference between the initial road surface recognition results corresponding to each sensor based on the original probability of the initial road surface recognition results corresponding to each sensor; and if the probability difference meets a preset condition, take the initial road surface recognition result corresponding to the maximum original probability as the road surface recognition result of the road where the vehicle is traveling.

[0172] The preset conditions include: the probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest; and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result. In this application, probability can be represented by confidence level.

[0173] In some embodiments, the initial road surface recognition result includes the original probabilities corresponding to the candidate road surface types. Determining the road surface recognition result of the road the vehicle is traveling on based on the initial road surface recognition results corresponding to each sensor can be achieved as follows: if the probability difference does not meet a preset condition, determine the comprehensive probability of each candidate road surface type based on the original probabilities corresponding to each candidate road surface type in the initial road surface recognition results corresponding to each sensor; and determine the road surface type of the road the vehicle is traveling on based on the comprehensive probability of each candidate road surface type.

[0174] In some embodiments, determining the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type can be achieved by: weighting and fusing the original probabilities corresponding to the same candidate road surface type in the initial road surface recognition results of each sensor to obtain the comprehensive probability of the candidate road surface type; and determining the road surface type corresponding to the maximum comprehensive probability as the road surface type of the vehicle's driving road.

[0175] For example, during weighted fusion, since the perception dimensions of each sensor are different, the initial road surface recognition results of each sensor need to be uniformly converted into the same data type, namely, the recognition probability vectors (initial probabilities) of various road surface types. The recognition probability vector corresponding to the tire noise sensor is [0.7 (snow)], the recognition probability vector corresponding to the vision sensor is [0.6 (asphalt)], and the recognition probability vector corresponding to the tactile sensor is [0.7 (snow)]. Assuming that the weights of vision, tactile, and tire noise are 0.3, 0.2, and 0.5 respectively, the combined probability of snow and asphalt can be calculated according to the weighted summation formula. The combined probability of snow is greater than that of asphalt, so snow is taken as the road surface type of the current vehicle driving road.

[0176] It should be noted that the weights of each sensor can be fixed preset values. These fixed preset values ​​can be determined using experimental data from Dalian. This method only requires weighted calculation based on fixed preset values. The algorithm is simple and suitable for vehicles with limited computing resources.

[0177] For example, in weighted fusion, the weights corresponding to the visual sensors are determined based on the visibility of the vehicle's environment. The weights corresponding to the tire noise sensors are determined based on at least one of the noise level of the vehicle's environment and the vehicle's motion state. The weights of the tactile sensors are determined based on the vehicle's motion state.

[0178] Since fixed preset values ​​cannot be adapted to all driving scenarios, and fixed weights will distort the road recognition results, the weights of each sensor can also be dynamically changed according to the vehicle's driving status.

[0179] Understandably, visual sensors are susceptible to interference from light and rain / fog. When visibility is low, the reliability of visual sensor recognition decreases. In such cases, the weight of the visual sensor can be dynamically reduced. For example, in scenarios where visibility is severely limited, such as heavy rain or fog, the weight of the visual sensor can be set to 0. In scenarios such as at night or in tunnels, the weight of the visual sensor can be appropriately reduced.

[0180] The core of road surface recognition based on tire noise sensors lies in analyzing the tire noise characteristics detected by the sensors to identify the road surface. However, when ambient noise is high, it can overlap with the tire noise signal, resulting in inaccurate tire noise characteristics and distorted road surface recognition results. Furthermore, when the vehicle is stationary, the tire noise sensor cannot collect tire noise signals, making it impossible to obtain road surface recognition results. In such cases, the weight corresponding to the tire noise sensor can be set to 0.

[0181] Similar to tire noise sensors, when the vehicle is stationary, tactile sensors cannot acquire the vehicle's motion parameters. In this case, the tactile sensors cannot obtain road surface recognition results. The weight of the tactile sensors can be set to 0.

[0182] In some embodiments, during weighted fusion, the weights corresponding to each sensor are determined as follows: based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined; based on the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor, the weights corresponding to each sensor are determined.

[0183] In some embodiments, the controller is configured to determine the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road on which the vehicle travels. This can be achieved by: determining the attention score of the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on an attention mechanism; and determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the attention score of each sensor.

[0184] In some embodiments, the controller is configured to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention score of each sensor. This can be achieved by normalizing the attention score of each sensor and determining the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

[0185] For example, such as Figure 4As shown, the road surface type of the road on which the vehicle travels can be determined by a dynamic weight update method based on an attention learning mechanism.

[0186] The inputs to the dynamic weight update method of the attention learning mechanism are: ① the initial road recognition results and confidence scores of the three senses at the current moment; ② the road recognition results and confidence scores of the three senses over the past n time steps; ③ the final road recognition results after the fusion of the three senses over the past n time steps.

[0187] The output of the dynamic weight update method of the attention learning mechanism is: the three-sense fusion road surface type recognition result at the current moment.

[0188] The implementation process of the dynamic weight update method of the attention learning mechanism is as follows: through the attention scoring mechanism, the attention scores of the initial road surface recognition results of each perception mode in the past n time steps and the recognition results of the three senses fusion in the past n time steps are calculated respectively, and the attention score a is calculated by formula (1).

[0189] a = q T Wk formula(1)

[0190] Where q represents the initial road surface recognition result of each perception method over the past n time steps, k represents the road surface recognition result of the three-sensory fusion over the past n time steps, W is an n×n weight matrix, obtained through offline training, and a represents the attention score of each perception method.

[0191] The attention scores of the three perception methods are normalized using a softmax layer, resulting in dynamic weights for each sensor. These dynamic weights are then multiplied by the sensor's confidence level (initial probability) and the probabilities of the same type are summed. The road type with the highest confidence level is the final road type result.

[0192] In some embodiments, the controller is further configured to: if the road surface types in the initial road surface recognition results are consistent, before using the initial road surface recognition results as the road surface recognition results of the road where the vehicle is traveling, filter the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor.

[0193] In some embodiments, the controller is configured to filter the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor. This can be achieved by deleting the initial road surface recognition results containing snow when the ambient temperature is greater than a preset temperature threshold, thereby obtaining the filtered initial road surface recognition results corresponding to each sensor.

[0194] like Figure 5As shown, the process of determining the road surface recognition result of the vehicle's route based on the initial road surface recognition results corresponding to each sensor may include the following steps:

[0195] S501. Obtain the ambient temperature, the road surface type and its confidence level in the initial road surface identification results of each sensor.

[0196] S502. Determine whether the ambient temperature is greater than the preset temperature threshold. If yes, proceed to S503; otherwise, proceed to S504.

[0197] S503. Exclude snow-covered road surface types; consider initial road surface identification results as snow-covered as invalid results.

[0198] S504. Are all valid initial road surface identification results consistent? If yes, proceed to S505; otherwise, proceed to S506.

[0199] S505. The initial road surface recognition result with the highest confidence level shall be used as the road surface recognition result.

[0200] S506. Are the differences in confidence levels between the initial road surface identification result with the highest confidence level and other initial road surface identification results greater than the preset confidence threshold? If yes, proceed to S505; otherwise, proceed to S507.

[0201] S507. The dynamic weight update method based on the attention learning mechanism determines the road surface recognition result of the vehicle driving road.

[0202] In some embodiments, the controller is configured to: smooth the overall probability of each candidate road surface type using a moving average function to determine the smoothed overall probability of each candidate road surface type; and determine the road surface type of the road on which the vehicle travels based on the smoothed overall probability of each candidate road surface type.

[0203] As one possible implementation, by setting sliding window parameters, the average value of the comprehensive probability of each candidate road surface type within each sliding window is determined, and the average value is used as the comprehensive probability of each road surface type after smoothing.

[0204] The average of the comprehensive probability of each road surface type within each sliding window can be calculated using either the simple averaging method or the exponential averaging method; no specific restrictions are imposed here.

[0205] Therefore, by smoothing the comprehensive probability, the impact of instantaneous fluctuations on road surface recognition results can be avoided, the risk of misjudgment can be reduced, and the accuracy of road surface type recognition for vehicles can be improved.

[0206] In some embodiments, the controller is further configured to: determine whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode when the current driving mode and the target driving mode are inconsistent.

[0207] Understandably, the purpose of designing different driving modes is to ensure that vehicles can safely and efficiently traverse different road surfaces; for example, snow mode corresponds to snow conditions, and mud mode corresponds to mud conditions. Therefore, the safety levels of different driving modes vary. When switching driving modes, directly switching to a lower-safety mode when the current road surface demands extremely high driving safety may introduce safety hazards.

[0208] In some embodiments, determining whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode can be implemented as follows: if the safety level of the current driving mode is lower than the safety level of the target driving mode, determine that the vehicle is switching from the current driving mode to the target driving mode; or, if the safety level of the current driving mode is higher than the safety level of the target driving mode, and the vehicle meets the driving mode switching conditions, determine that the vehicle is switching from the current driving mode to the target driving mode.

[0209] The driving mode switching conditions can be either user confirmation or whether the duration of the current driving mode exceeds a preset time threshold.

[0210] For example, driving modes are divided into multiple levels based on safety and aggression. The lower the level, the safer it is, and the higher the level, the more aggressive it is. When switching from a lower level to a higher level, more stringent conditions must be met before the switch can be made.

[0211] Therefore, by comparing the safety levels of the current driving mode and the target driving mode to determine whether to switch, the design principle of prioritizing vehicle safety is realized, thereby avoiding the risk of switching from a high safety level mode to a low safety level mode and improving the safety of vehicle driving.

[0212] In some embodiments, the controller is further configured to determine a target driving mode based on road surface recognition results of the road where the vehicle is traveling.

[0213] In one possible implementation, a neural network model is trained using sample road surface recognition results. The neural network model can then autonomously generate a target driving mode based on the current road surface recognition results.

[0214] For example, when the road surface identification result of the current vehicle driving road is snow, the target driving mode is: control the vehicle to be in snow driving mode, activate the traction control system for anti-skid control, and adjust the torque of each wheel according to the road surface characteristics - adhesion coefficient - based on the road surface identification result.

[0215] Therefore, by determining the target driving mode based on the current road surface recognition results, the vehicle can accurately adapt to different road surfaces, thereby improving driving safety and comfort.

[0216] In some embodiments, the controller is configured to: determine the matching score between the current road surface recognition result and each driving mode based on the road surface type and / or road surface characteristics; determine the pre-selected driving mode and the priority of the pre-selected driving mode based on the matching score; and select the one with the highest priority as the target driving mode.

[0217] For example, the matching score can be represented by a vector. The matching degree between all road surface types and all driving modes can be represented as a two-dimensional vector table. The matching degree between each road surface feature and each driving mode can be represented as a 3×m two-dimensional vector table, where 3 represents that there are three levels of adhesion coefficient and m represents the number of driving modes. By looking up the vector table, the matching score between the current road surface recognition result and each driving mode can be determined.

[0218] Assuming that the matching score for each driving mode is S, and D is the preset score redundancy, representing the acceptable error size, all driving modes with scores higher than SD are selected as pre-selected driving modes.

[0219] In some embodiments, after determining the target driving mode, the controller is configured to: determine whether the current driving mode and the target driving mode are the same; if they are the same, maintain the current driving mode; if they are different, determine that the priority of the target driving mode is higher than the priority of the current driving mode; if so, switch the current driving mode to the target driving mode; if not, determine whether the duration of the current driving mode is greater than a preset value; if so, the switch can be made; otherwise, maintain the current driving mode.

[0220] like Figure 6 As shown, the process of determining and switching the target driving mode can be as follows:

[0221] S601. Obtain road surface type and road surface characteristics.

[0222] S602. Calculate the matching score of each driving mode by referring to the table based on the road surface type and road surface characteristics.

[0223] S603. Determine the driving mode priority based on the safety of the driving mode.

[0224] S604. Based on the matching score, determine the candidate driving modes and take the candidate driving mode with the highest priority as the target driving mode.

[0225] S605. Determine whether the target driving mode and the current driving mode are the same. If yes, proceed to S606; otherwise, proceed to S607.

[0226] S606, Maintain current driving mode.

[0227] S607. Is the priority of the target driving mode higher than that of the current driving mode? If yes, proceed to S608; otherwise, proceed to S609.

[0228] S608, switch to target driving mode.

[0229] S609. Does the duration of the current driving mode exceed the preset time threshold? If yes, execute S608; otherwise, execute S606.

[0230] In some embodiments, the controller is further configured to control the actuator to perform adjustment actions corresponding to the target driving mode.

[0231] As one possible implementation, the controller issues commands to each actuator according to the target driving mode, so that each actuator works in accordance with the target driving mode, and the actuator responds to the command after receiving the command.

[0232] The following section details the process by which each actuator performs the adjustment actions corresponding to the target driving mode.

[0233] (1) Powertrain subsystem (for gasoline vehicles, it is the engine controller; for pure electric vehicles, it is the motor controller): The powertrain subsystem can improve stability by distributing torque vectors according to wheel slippage. For example, it reduces torque to prevent slippage on snowy roads and increases torque to improve power in paved road driving mode; on sandy or muddy roads, the system can identify when wheels are stuck and help the vehicle get out of trouble by outputting differential torque.

[0234] (2) Braking Subsystem: This system regulates the wheel-end torque of each wheel by applying braking force. It adjusts the braking force response to the brake pedal according to different driving modes. Paved roads allow for greater braking force to shorten braking distance; braking force is only adjusted when the wheels lock up. On icy or snowy roads, where the coefficient of friction is low and slippage and loss of control are easy, the maximum braking force needs to be limited to prevent wheel lockup. On soft surfaces, tires are prone to sinking or slipping; the system can identify sinking or slipping tires and avoid excessive braking force that could cause the tires to sink even deeper.

[0235] (3) Active Suspension Subsystem: This subsystem adaptively adjusts suspension stiffness, damping, height, and other parameters in real time to improve vehicle stability, passability, and ride comfort under different terrain conditions. When traveling at high speeds on paved roads, the suspension height is lowered to improve vehicle dynamics and enhance stability. On complex road conditions such as potholes and gravel roads, the suspension height is increased to improve vehicle passability and reduce the risk of damage to the vehicle's undercarriage. On slippery or icy roads with low coefficients of friction, the suspension damping is reduced to allow the wheels to adapt more freely to road undulations and improve grip.

[0236] (4) Steering subsystem: Based on the terrain recognition results, the steering effort, steering response speed and steering ratio are adaptively adjusted in real time. For example, in paved road driving mode, the road feel feedback is enhanced and the steering ratio is increased to enhance steering sensitivity; on low-adhesion surfaces such as water accumulation and snow, the steering assist motor applies resistance to suppress excessive steering wheel rotation.

[0237] (5) Electronic differential: The core function of the electronic differential is to dynamically adjust the speed difference between the left and right wheels. By simulating the function of a mechanical differential lock through electronic control, it adjusts the power distribution in real time according to the terrain, suppresses slippage, and improves the passability and stability in complex road conditions. For example, on paved roads, there is no need to lock it to avoid damage to components and handling risks; when off-roading and bumpy, locking it can prevent the wheels from lifting off the ground and losing traction; when climbing mountains, the differential lock needs to be locked to deal with cross-axle conditions and ensure sufficient traction.

[0238] In some embodiments, the intelligent driving domain controller is configured to: receive sensor data sent by sensors, and determine the road surface recognition result of the road where the vehicle is traveling based on the sensor data sent by multiple sensors.

[0239] The road surface recognition process of the intelligent driving domain controller can be referred to the description in the above embodiments, and will not be repeated here.

[0240] Understandably, the intelligent driving domain controller, due to its powerful computing capabilities, can serve as the core carrier of the road recognition algorithm, ensuring that the algorithm can process sensor data in real time and output accurate road recognition results.

[0241] In some embodiments, the intelligent driving domain controller is further configured to send road surface recognition results of the road where the vehicle is traveling to the vehicle controller.

[0242] Understandably, because road surface recognition results affect multiple actuators in the vehicle, which may belong to different control domains (power domain, braking domain), the vehicle controller needs to make a comprehensive judgment. Furthermore, the vehicle controller generates unified control commands based on the road surface recognition results, controlling the various actuators to work collaboratively, enabling the vehicle to accurately adapt to the current road surface, thereby ensuring a balance in safety, power, and comfort.

[0243] In some embodiments, the intelligent driving domain controller can not only perform road surface recognition, but also make decisions on driving modes. However, since the intelligent driving domain controller does not communicate directly with the various actuators of the vehicle, it needs to pass the target driving mode to the vehicle controller, which is closer to the actuators, for execution.

[0244] Therefore, the intelligent driving domain controller is also configured to: determine the target driving mode based on the road surface recognition results of the road where the vehicle is traveling, and send the target driving mode to the vehicle controller.

[0245] The vehicle controller is configured to control the actuators to perform adjustment actions corresponding to the target driving mode.

[0246] Among them, the vehicle controller, as the control center of the entire vehicle, is responsible for coordinating the operation of the vehicle's controllers to ensure that the target driving mode is implemented.

[0247] In other embodiments, the intelligent driving domain controller may only execute the road surface recognition algorithm, while the driving mode decision is executed by the vehicle controller. Since the vehicle controller can comprehensively judge the vehicle's state by combining the various actuators of the vehicle, making the driving mode decision by the vehicle controller is more in line with the actual capabilities of the vehicle and avoids a disconnect between decision-making and execution.

[0248] In some embodiments, the human-computer interaction device is configured to generate a road surface recognition command and send the road surface recognition command to the controller.

[0249] The controller is configured to, in response to a road surface recognition command, determine the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors.

[0250] Understandably, human-computer interaction devices, as devices that transform user intentions into control commands, generate road surface recognition commands by parsing the user's operational intentions. The controller responds to the road surface recognition commands to perform road surface recognition, and then controls the vehicle's movement state. This realizes the process from user needs to controller execution to result feedback to the vehicle, allowing users to participate in the operation of the vehicle and increasing the user's driving experience.

[0251] As one possible implementation, the road surface recognition command is triggered in at least one of the following ways: receiving a user's touch operation on the display interface of the in-vehicle display device; receiving a user's trigger operation on the road surface recognition button; or receiving a sound signal from the user to perform road surface recognition.

[0252] For example, a virtual "road surface recognition" button is provided in the vehicle's central control screen, instrument panel, or other interactive interface, allowing users to trigger road surface recognition commands through touch operations such as clicking or long-pressing. Physical buttons are also provided on the steering wheel and center console, allowing users to trigger road surface recognition commands through touch operations such as pressing the buttons or rotating the knobs.

[0253] In some embodiments, the controller is also configured to send road surface recognition results to the human-machine interface device.

[0254] The human-computer interaction device is also configured to receive and display the road surface recognition results sent by the controller.

[0255] For example, the road recognition results can be displayed in the form of text on the in-vehicle central control screen, such as: "Current road surface: snow, it is recommended to turn on snow mode"; the road recognition results can also be displayed in the form of icons on the instrument panel, such as: snowflake icon + road features; the road recognition results can also be displayed through the head-up display (HUD), such as: "Current road surface: snow".

[0256] Thus, by displaying the road recognition results through a human-computer interaction device, a closed loop is completed, from the user initiating the road recognition command to the vehicle control system recognizing the results and then displaying the results. This allows the user's operation to receive feedback, and this closed-loop feedback increases trust in the vehicle's intelligent driving. After receiving the road recognition results, the user can proactively adjust their driving behavior and reduce the risk of accidents.

[0257] In some embodiments, the controller is further configured to send an information display instruction carrying prompt information to the human-machine interface device when the vehicle's current driving mode and target driving mode are inconsistent; the prompt information includes: road recognition results and target driving mode.

[0258] The human-computer interaction device is also configured to: receive and parse information display instructions to obtain prompts; and display the prompts.

[0259] As one possible implementation, the prompt message can be displayed in at least one of the following ways: by displaying the prompt message on the display interface of the in-vehicle display device; or by outputting the prompt message through the in-vehicle audio device.

[0260] When the vehicle's current driving mode and the target driving mode are inconsistent, prompts are delivered to the user through human-computer interaction, thereby conveying the system's decisions to the user and making the intelligent driving decisions of the vehicle control system transparent.

[0261] In some embodiments, the human-machine interface device is further configured to generate a confirmation command for the target driving mode.

[0262] The controller is also configured to control the vehicle to drive based on the target driving mode in response to a confirmation command for the target driving mode.

[0263] As one possible implementation, the confirmation command for the target driving mode is triggered by at least one of the following methods: receiving a driving mode confirmation operation from the user on the display interface of the in-vehicle display device; receiving a trigger operation from the user on the driving mode confirmation button; or receiving an audio signal from the user confirming the target driving mode.

[0264] Understandably, by acquiring user interaction data, it is possible to determine whether the vehicle is driving based on the target driving mode, thus ensuring the user's ultimate decision-making power over vehicle control. At the same time, by actively participating in driving decisions rather than passively accepting system commands, users can enhance their trust in intelligent driving functions.

[0265] The following is combined with Figure 3 This application describes the working process of the vehicle control system.

[0266] Users can activate the road surface recognition function via the central control panel or the driving mode selection button. Multiple sensor data are provided by the vision sensor 1022, tire noise sensor 1021, and tactile sensor 1023, and the sensor data is sent to the intelligent driving domain controller 1011.

[0267] The fusion processing of sensor data and the road surface recognition algorithm are integrated into the intelligent driving domain controller 1011. The intelligent driving domain controller 1011 determines the road surface recognition result based on multiple sensor data, and determines the target driving mode based on the road surface recognition result and sends it to the in-vehicle multimedia host and the vehicle controller 1012. The multimedia host displays the road surface recognition result and the target driving mode through the instrument cluster. The vehicle controller 1012 determines whether to switch the driving mode based on the road surface recognition result and the target driving mode.

[0268] The sensor data acquisition process includes: acquiring road images in front of the vehicle via a camera; acquiring tire noise signals needed for road surface identification via a microphone; and obtaining tactile sensor data through inertial navigation sensors, suspension height sensors, wheel speed sensors, wheel speed sensors, torque sensors, steering wheel angle sensors, brake pedal sensors, and brake torque sensors.

[0269] The road surface recognition process includes: the intelligent driving domain controller 1011 receiving, analyzing, and processing road images, tire noise signals, tactile sensor data, etc., to determine the road surface recognition result and the target driving mode. The vehicle controller 1012 determines whether to switch the driving mode based on the safety level of the driving mode and the driving mode switching conditions.

[0270] The human-machine interaction process includes: the user selects whether to enable the road surface recognition function via the central control panel or the driving mode selection button. The intelligent driving domain controller outputs the road surface recognition results and the target driving mode to the multimedia host. The multimedia host displays relevant information about all-terrain adaptive control to the user through the screen and instrument cluster, including: the current driving mode, the current road surface type, and the current road surface characteristics.

[0271] The actuator works as follows: The vehicle controller 1012 issues commands to the actuators according to the target driving mode, so that each actuator works in accordance with the target driving mode, thereby improving the vehicle's safety, passability and smoothness under different road conditions.

[0272] In some embodiments, the road surface recognition function can be activated when the vehicle is in all-terrain driving mode. After activating all-terrain driving mode, the vehicle automatically switches driving modes and displays the road surface recognition results by recognizing the road surface, thereby alleviating the user's anxiety and increasing the user's confidence in navigating complex road surfaces.

[0273] In some embodiments, the road surface recognition function can also be triggered when the vehicle activates the road surface preview function. By previewing the road surface ahead, the driver can prepare in advance and achieve safer and more efficient driving control in complex road conditions.

[0274] In some embodiments, the road surface recognition function can also be triggered when the vehicle activates the anti-skid control function. Since whether a vehicle will skid is closely related to the road surface type and characteristics, after the vehicle activates the anti-skid control function, by recognizing the road surface, the anti-skid control function can more accurately adapt to the current road surface characteristics, thereby improving the targeting and effectiveness of the anti-skid effect.

[0275] The following is combined with Figure 7 This section describes the control signal interaction process in a vehicle control system.

[0276] Users can select whether to enable the road surface recognition function via the central control panel or the driving mode selection button (physical). User input signals are received by the vehicle's multimedia host via the CAN bus and then sent to the intelligent driving domain controller via Ethernet to control the activation or deactivation of the road surface recognition function.

[0277] The intelligent driving domain controller receives and analyzes road images from cameras, tire noise signals from tire noise sensors, and sensor data from tactile sensors (power subsystem). It executes road surface recognition algorithms to identify road surface types and features, and sends the road surface recognition results and user prompts to the instrument cluster for display.

[0278] At the same time, the intelligent driving domain controller determines the target driving mode corresponding to the road recognition result and controls the actuators (electronic differential, power subsystem, braking subsystem, steering subsystem, active suspension subsystem) to perform the adjustment actions corresponding to the target driving mode.

[0279] The multimedia host is the display carrier of the vehicle control system. It displays driving mode function options through the display screen and instrument cluster. The driver can activate the road recognition function through the function access button on the touch screen. The multimedia host sends a road recognition function activation signal to the intelligent driving domain controller. The intelligent driving domain controller outputs the road recognition result and the target driving mode. The vehicle controller determines whether to switch driving modes and transmits the road recognition result and the target driving mode to the multimedia host. The multimedia host displays relevant information about all-terrain adaptive control to the user through the screen and instrument cluster, including: the current driving mode, the current road type, and the current road characteristics.

[0280] Among them, the camera, tire noise sensor, and multimedia host communicate with the intelligent driving domain controller via Ethernet, while other communication methods are all CAN bus communication.

[0281] like Figure 8 As shown, this application provides a vehicle control method applied to the aforementioned vehicle control system, comprising:

[0282] S801: Acquire sensor data from multiple sensors.

[0283] Among them, multiple sensors include at least: tire noise sensor.

[0284] S802. Based on sensor data sent by multiple sensors, determine the road surface recognition result of the road where the vehicle is traveling.

[0285] In some embodiments, the road surface identification results include at least one of the following: road surface type and road surface features.

[0286] In some embodiments, the above-mentioned road surface characteristics include at least one of the following: softness, slope, rolling resistance, smoothness, adhesion coefficient, and slip ratio.

[0287] In some embodiments, the above-mentioned road surface types include at least one of the following: snow, mud, grassland, sand, mountain, rock, wading, and asphalt.

[0288] In some embodiments, determining the road surface identification result of the road surface on which the vehicle is traveling based on sensor data sent by multiple sensors includes: determining the softness of the road surface on which the vehicle is traveling based on sensor data sent by a tire noise sensor.

[0289] In some embodiments, the sensor further includes at least one of a vision sensor and a tactile sensor.

[0290] In some embodiments, determining the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors includes: determining the initial road surface recognition result corresponding to each sensor based on the sensor data sent by each sensor; and determining the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor.

[0291] In some embodiments, the initial road surface identification result includes the original probability corresponding to each candidate road surface type; the road surface identification result includes the road surface type of the road on which the vehicle travels.

[0292] In some embodiments, determining the road surface identification result of the road where the vehicle is traveling based on multiple sensor data includes: if the road surface type in the initial road surface identification result is consistent, using the initial road surface identification result as the road surface identification result of the road where the vehicle is traveling.

[0293] In some embodiments, the above-mentioned determination of the road surface recognition result of the road where the vehicle is traveling based on multiple sensor data further includes: when the road surface types in the initial road surface recognition results are inconsistent, determining the probability difference between the initial road surface recognition results corresponding to each sensor based on the original probability of the initial road surface recognition results corresponding to each sensor; and when the probability difference meets a preset condition, taking the initial road surface recognition result corresponding to the maximum confidence level as the road surface recognition result of the road where the vehicle is traveling.

[0294] In some embodiments, the preset conditions include: the probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest; and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

[0295] In some embodiments, the above-mentioned determination of the road surface recognition result of the vehicle's driving road based on the initial road surface recognition result corresponding to each sensor further includes: determining the comprehensive probability of each candidate road surface type based on the original probability corresponding to each candidate road surface type in the initial road surface recognition result corresponding to each sensor; and determining the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type.

[0296] In some embodiments, determining the comprehensive probability of each candidate road type based on the original probabilities of each candidate road type in the initial road recognition results corresponding to each sensor includes: weighting and fusing the original probabilities of the same candidate road type in the initial road recognition results corresponding to each sensor to obtain the comprehensive probability of the candidate road type.

[0297] In some embodiments, the weights corresponding to the visual sensors in the weighted fusion process are determined based on the visibility of the vehicle's environment.

[0298] In some embodiments, during weighted fusion, the weights corresponding to the tire noise sensors are determined based on at least one of the noise of the vehicle's environment and the vehicle's motion state.

[0299] In some embodiments, the weights of the tactile sensors are determined based on the vehicle's motion state during weighted fusion.

[0300] In some embodiments, during the weighted fusion process, the weights corresponding to each sensor are determined as follows: based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined; based on the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor, the weights corresponding to each sensor are determined.

[0301] In some embodiments, determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road on which the vehicle travels includes: determining the attention score between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on an attention mechanism; and determining the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the attention score of each sensor.

[0302] In some embodiments, determining the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention scores of each sensor includes: normalizing the attention scores of each sensor to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

[0303] In some embodiments, the method further includes: if the road surface types in the initial road surface recognition results are consistent, before using the initial road surface recognition results as the road surface recognition results of the road where the vehicle is traveling, filtering the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor.

[0304] In some embodiments, the above-mentioned filtering of the initial road surface recognition results corresponding to each sensor based on ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor includes: deleting the initial road surface recognition results containing snow when the ambient temperature is greater than a preset temperature threshold, thereby obtaining the filtered initial road surface recognition results corresponding to each sensor.

[0305] In some embodiments, determining the road surface type of the vehicle's travel path based on the comprehensive probability of each candidate road surface type includes: smoothing the comprehensive probability of each road surface type using a moving average function to determine the smoothed comprehensive probability of each road surface type; and determining the road surface type of the vehicle's travel path based on the smoothed comprehensive probability of each road surface type.

[0306] In some embodiments, determining the road surface type of the vehicle's driving road based on the comprehensive probability of each candidate road surface type includes: determining the road surface type corresponding to the maximum comprehensive probability as the road surface type of the vehicle's driving road.

[0307] In some embodiments, determining the road surface identification result of the road where the vehicle is traveling based on sensor data sent by multiple sensors includes: determining the tire noise characteristics of each wheel based on sensor data sent by the tire noise sensors of each wheel; and determining the road surface identification result of the road where the vehicle is traveling based on the tire noise characteristics of each wheel.

[0308] In some embodiments, determining the road surface recognition result of the road where the vehicle is traveling based on the tire noise characteristics of each wheel includes: determining the road surface recognition result of each wheel based on the tire noise characteristics of each wheel; and determining the road surface recognition result of the road where the vehicle is traveling based on the road surface recognition result of each wheel.

[0309] In some embodiments, determining the road surface recognition result of each wheel based on the tire noise characteristics of each wheel includes: fusing the tire noise characteristics of each wheel to obtain the fused tire noise characteristics of each wheel; and determining the road surface recognition result of each wheel based on the fused tire noise characteristics of each wheel.

[0310] In some embodiments, the above-mentioned fusion of the tire noise features of each wheel to obtain the fused tire noise features of each wheel includes: feature splicing of the tire noise features of each wheel to obtain the fused tire noise features of each wheel.

[0311] In some embodiments, the above-mentioned tire noise characteristics include at least one of the following: statistical characteristics, Mel time-frequency characteristics, and wavelet transform time-frequency characteristics.

[0312] In some embodiments, the method further includes: when the current driving mode and the target driving mode of the vehicle are inconsistent, determining whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode.

[0313] In some embodiments, determining whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode includes: determining that the vehicle switches from the current driving mode to the target driving mode when the safety level of the current driving mode is lower than the safety level of the target driving mode; or, determining that the vehicle switches from the current driving mode to the target driving mode when the safety level of the current driving mode is higher than the safety level of the target driving mode and the vehicle meets the driving mode switching conditions.

[0314] In some embodiments, the method further includes: determining a target driving mode based on the road surface recognition results of the road where the vehicle is traveling.

[0315] In some embodiments, the method further includes: controlling the actuator to perform adjustment actions corresponding to the target driving mode.

[0316] In some embodiments, the method further includes: in response to a road surface recognition command, determining a road surface recognition result for the road surface on which the vehicle is traveling based on sensor data sent by multiple sensors.

[0317] In some embodiments, the road surface recognition command is triggered by at least one of the following methods: receiving a touch operation from a user on the display interface of an in-vehicle display device; receiving a trigger operation from a user on the road surface recognition button; or receiving a sound signal from a user carrying out road surface recognition.

[0318] In some embodiments, the above method further includes: displaying the road surface recognition results.

[0319] In some embodiments, the method further includes: displaying a prompt message when the vehicle's current driving mode and the target driving mode are inconsistent; the prompt message includes: road surface recognition results and the target driving mode.

[0320] In some embodiments, the above-mentioned prompt information is displayed in at least one of the following ways: displaying the prompt information through the display interface of the in-vehicle display device; or outputting the prompt information through the in-vehicle sound device.

[0321] In some embodiments, the method further includes: controlling the vehicle to drive based on the target driving mode in response to a confirmation command for the target driving mode.

[0322] In some embodiments, the confirmation command for the target driving mode is triggered by at least one of the following methods: receiving a driving mode confirmation operation from a user on the display interface of the in-vehicle display device; receiving a user's trigger operation on the driving mode confirmation button; or receiving a sound signal from the user confirming the target driving mode.

[0323] In some embodiments, determining the road surface recognition result of the road where the vehicle is currently traveling based on sensor data sent by multiple sensors includes: segmenting the sensor data sent by the vision sensor to obtain multiple regional sub-images; the sensor data includes: road surface images of the road where the vehicle is currently traveling; filtering the categories of the multiple regional sub-images to determine regional sub-images of the category of road surface; the category of the regional sub-images includes at least one of the following: road surface, obstacle, lane line; and determining the road surface recognition result of the road where the vehicle is currently traveling based on the regional sub-images of the category of road surface.

[0324] In some embodiments, the above-mentioned segmentation of the road surface image of the road currently being traveled by the vehicle to obtain multiple regional sub-images includes: performing semantic segmentation of the road surface image based on a neural network model to determine the category of each pixel in the road surface image; and dividing the road surface image into multiple regional sub-images based on the category of each pixel.

[0325] In some embodiments, determining the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors includes: extracting features from sensor data sent by tactile sensors to obtain the vehicle's motion features; determining the road surface features of the road where the vehicle is currently traveling based on the motion features; and determining the road surface recognition result of the road where the vehicle is currently traveling based on the road surface features.

[0326] In some embodiments, the above-mentioned motion characteristics include at least one of the following: four-wheel load, vehicle speed, gradient, rolling resistance, slip ratio, coefficient of adhesion, and road surface smoothness.

[0327] The implementation process of the vehicle control method can be referred to the description of the vehicle control system above, and will not be repeated here.

[0328] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the vehicle control device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0329] This application embodiment can, according to the above method, exemplarily divide a vehicle control device into functional modules. For example, the vehicle control device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0330] In an exemplary embodiment, this application also provides a vehicle control device for performing the above-described vehicle control method.

[0331] like Figure 9 As shown, the vehicle control device 900 includes an acquisition unit 901 and a processing unit 902.

[0332] Acquisition unit 901 is used to acquire sensor data from multiple sensors; the sensors include at least: a tire noise sensor;

[0333] The processing unit 902 is used to determine the road surface recognition result of the road where the vehicle is traveling based on sensor data sent by multiple sensors.

[0334] In some embodiments, the road surface identification results include at least one of the following: road surface type and road surface features.

[0335] In some embodiments, the above-mentioned road surface characteristics include at least one of the following: softness, slope, rolling resistance, smoothness, adhesion coefficient, and slip ratio.

[0336] In some embodiments, the above-mentioned road surface types include at least one of the following: snow, mud, grassland, sand, mountain, rock, wading, and asphalt.

[0337] In some embodiments, the processing unit 902 is specifically used to determine the softness of the road on which the vehicle is traveling based on sensor data sent by the tire noise sensor.

[0338] In some embodiments, the sensor further includes at least one of a vision sensor and a tactile sensor.

[0339] In some embodiments, the processing unit 902 is specifically used to determine the initial road surface recognition result corresponding to each sensor based on the sensor data sent by each sensor; and to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each sensor.

[0340] In some embodiments, the initial road surface identification result includes the original probability corresponding to each candidate road surface type; the road surface identification result includes the road surface type of the road on which the vehicle travels.

[0341] In some embodiments, the processing unit 902 is specifically used to use the initial road surface recognition result as the road surface recognition result of the road where the vehicle is traveling, when the road surface type in the initial road surface recognition result is consistent.

[0342] In some embodiments, the processing unit 902 is further configured to determine the probability difference between the initial road surface recognition results of each sensor based on the original probability of the initial road surface recognition results of each sensor when the road surface types in the initial road surface recognition results are inconsistent; and to take the initial road surface recognition result corresponding to the maximum confidence value as the road surface recognition result of the road where the vehicle is traveling when the probability difference meets the preset conditions.

[0343] In some embodiments, the preset conditions include: the probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest; and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

[0344] In some embodiments, the processing unit 902 is further configured to determine the comprehensive probability of each candidate road surface type based on the original probability of each candidate road surface type in the initial road surface recognition results corresponding to each sensor; and to determine the road surface type of the road on which the vehicle travels based on the comprehensive probability of each candidate road surface type.

[0345] In some embodiments, the processing unit 902 is specifically used to perform weighted fusion of the original probabilities corresponding to the same candidate road surface type in the initial road surface recognition results of each sensor to obtain the comprehensive probability of the candidate road surface type.

[0346] In some embodiments, the weights corresponding to the visual sensors in the weighted fusion process are determined based on the visibility of the vehicle's environment.

[0347] In some embodiments, during weighted fusion, the weights corresponding to the tire noise sensors are determined based on at least one of the noise of the vehicle's environment and the vehicle's motion state.

[0348] In some embodiments, the weights of the tactile sensors are determined based on the vehicle's motion state during weighted fusion.

[0349] In some embodiments, the processing unit 902 is specifically used to determine the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels; and to determine the weight corresponding to each sensor based on the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor.

[0350] In some embodiments, the processing unit 902 is specifically used to determine the attention score of the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention mechanism; and to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention score of each sensor.

[0351] In some embodiments, the processing unit 902 is specifically used to normalize the attention scores of each sensor and determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

[0352] In some embodiments, the processing unit 902 is further configured to, before using the initial road surface recognition result as the road surface recognition result of the road where the road surface type is consistent in the initial road surface recognition result, filter the initial road surface recognition results corresponding to each sensor based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor.

[0353] In some embodiments, the processing unit 902 is specifically used to delete the initial road surface recognition results containing snow when the ambient temperature is greater than a preset temperature threshold, and obtain the initial road surface recognition results corresponding to each sensor after filtering.

[0354] In some embodiments, the processing unit 902 is specifically used to smooth the comprehensive probability of each road surface type using a moving average function, determine the comprehensive probability of each road surface type after smoothing, and determine the road surface type of the road on which the vehicle travels based on the comprehensive probability of each road surface type after smoothing.

[0355] In some embodiments, the processing unit 902 is specifically used to determine the road surface type corresponding to the maximum comprehensive probability as the road surface type of the vehicle driving road.

[0356] In some embodiments, the processing unit 902 is specifically used to determine the tire noise characteristics of each wheel based on the sensor data sent by the tire noise sensors of each wheel; and to determine the road surface identification result of the road where the vehicle is traveling based on the tire noise characteristics of each wheel.

[0357] In some embodiments, the processing unit 902 is specifically used to determine the road surface recognition result of each wheel based on the tire noise characteristics of each wheel; and to determine the road surface recognition result of the road where the vehicle is traveling based on the road surface recognition result of each wheel.

[0358] In some embodiments, the processing unit 902 is specifically used to fuse the tire noise features of each wheel to obtain the fused tire noise features of each wheel; and to determine the road surface recognition result of each wheel based on the fused tire noise features of each wheel.

[0359] In some embodiments, the processing unit 902 is specifically used to perform feature splicing on the tire noise features of each wheel to obtain the fused tire noise features of each wheel.

[0360] In some embodiments, the above-mentioned tire noise characteristics include at least one of the following: statistical characteristics, Mel time-frequency characteristics, and wavelet transform time-frequency characteristics.

[0361] In some embodiments, the processing unit 902 is further configured to determine whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode when the current driving mode and the target driving mode of the vehicle are inconsistent.

[0362] In some embodiments, the processing unit 902 is specifically configured to determine that the vehicle will switch from the current driving mode to the target driving mode when the safety level of the current driving mode is lower than the safety level of the target driving mode; or, when the safety level of the current driving mode is higher than the safety level of the target driving mode and the vehicle meets the driving mode switching conditions, determine that the vehicle will switch from the current driving mode to the target driving mode.

[0363] In some embodiments, the processing unit 902 is further configured to determine a target driving mode based on the road surface recognition results of the road where the vehicle is traveling.

[0364] In some embodiments, the processing unit 902 is further configured to control the actuator to perform adjustment actions corresponding to the target driving mode.

[0365] In some embodiments, the processing unit 902 is further configured to, in response to a road surface recognition command, determine the road surface recognition result of the road surface on which the vehicle is traveling based on sensor data sent by multiple sensors.

[0366] In some embodiments, the road surface recognition command is triggered by at least one of the following methods: receiving a touch operation from a user on the display interface of an in-vehicle display device; receiving a trigger operation from a user on the road surface recognition button; or receiving a sound signal from a user carrying out road surface recognition.

[0367] In some embodiments, the processing unit 902 described above is also used to display the road surface recognition results.

[0368] In some embodiments, the processing unit 902 is further configured to display a prompt message when the vehicle’s current driving mode and the target driving mode are inconsistent; the prompt message includes: road recognition results and the target driving mode.

[0369] In some embodiments, the above-mentioned prompt information is displayed in at least one of the following ways: displaying the prompt information through the display interface of the in-vehicle display device; or outputting the prompt information through the in-vehicle sound device.

[0370] In some embodiments, the processing unit 902 is further configured to control the vehicle to drive based on the target driving mode in response to a confirmation command for the target driving mode.

[0371] In some embodiments, the confirmation command for the target driving mode is triggered by at least one of the following methods: receiving a driving mode confirmation operation from a user on the display interface of the in-vehicle display device; receiving a user's trigger operation on the driving mode confirmation button; or receiving a sound signal from the user confirming the target driving mode.

[0372] In some embodiments, the processing unit 902 is specifically used to segment sensor data sent by the vision sensor to obtain multiple regional sub-images; the sensor data includes: a road surface image of the road currently being driven by the vehicle; filtering the categories of the multiple regional sub-images to determine the regional sub-images of the category of road surface; the category of the regional sub-images includes at least one of the following: road surface, obstacle, lane line; and based on the regional sub-images of the category of road surface, determining the road surface recognition result of the road currently being driven by the vehicle.

[0373] In some embodiments, the processing unit 902 is specifically used to perform semantic segmentation on the road surface image based on a neural network model, determine the category of each pixel in the road surface image, and divide the road surface image into multiple regional sub-images based on the category of each pixel.

[0374] In some embodiments, the processing unit 902 is specifically used to extract features from the sensor data sent by the tactile sensor to obtain the motion features of the vehicle; based on the motion features, determine the road surface features of the road surface on which the vehicle is currently traveling; and based on the road surface features, determine the road surface recognition result of the road surface on which the vehicle is currently traveling.

[0375] In some embodiments, the above-mentioned motion characteristics include at least one of the following: four-wheel load, vehicle speed, gradient, rolling resistance, slip ratio, coefficient of adhesion, and road surface smoothness.

[0376] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 includes, but is not limited to, a processor 1001 and a memory 1002.

[0377] The memory 1002 described above is used to store the executable instructions of the processor 1001. It is understood that the processor 1001 is configured to execute instructions to implement the vehicle control method in the above embodiments.

[0378] It should be noted that those skilled in the art will understand that Figure 10 The device structure shown does not constitute a limitation on the electronic device; the electronic device 1000 may include, but is not limited to, the following: Figure 10 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0379] The processor 1001 is the control center of the device, connecting various parts of the device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1002, and by calling data stored in the memory 1002, thereby providing overall monitoring of the device. The processor 1001 may include one or more processing units. Optionally, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1001.

[0380] The memory 1002 can be used to store software programs and various data. The memory 1002 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0381] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1002 including instructions, which can be executed by a processor 1001 of an electronic device 1000 to implement the methods in the above embodiments.

[0382] In actual implementation, Figure 9 The acquisition unit 901 and processing unit 902 in the middle can be made by Figure 10 The processor 1001 calls the computer program stored in the memory 1002 to implement the process. The specific execution process can be found in the description of the method section in the previous embodiment, and will not be repeated here.

[0383] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0384] In an exemplary embodiment, this application also provides a vehicle, including the vehicle control device, the vehicle control system, the electronic device, or the computer-readable storage medium described above.

[0385] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by the processor 1001 of the electronic device 1000 to perform the methods described above.

[0386] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of an electronic device, they implement the various processes of the above method embodiments and achieve the same technical effect as the above method. To avoid repetition, they will not be described again here.

[0387] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0388] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0389] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the classified units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0390] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0391] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or a complete or partial classification of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0392] In the description of the embodiments of this application, specific features, structures, materials or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0393] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A vehicle control system, characterized in that, The vehicle control system includes: The controller and a plurality of sensors connected to the controller, the plurality of sensors including at least: a tire noise sensor; The controller is configured to: receive sensor data sent by the multiple sensors; and determine the road surface recognition result of the road where the vehicle is traveling based on the sensor data sent by the multiple sensors.

2. The system according to claim 1, characterized in that, The road surface identification results include at least one of the following: road surface type and road surface features.

3. The system according to claim 2, characterized in that, The road surface characteristics include at least one of the following: softness, slope, rolling resistance, smoothness, adhesion coefficient, and slip ratio.

4. The system according to claim 2, characterized in that, The road surface type includes at least one of the following: snow, mud, grassland, sand, mountain, rock, water crossing, and asphalt.

5. The system according to claim 3, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on sensor data sent by the multiple sensors, including: The softness of the road on which the vehicle is traveling is determined based on the sensor data sent by the tire noise sensor.

6. The system according to any one of claims 1 to 5, characterized in that, The plurality of sensors also includes at least one of a vision sensor and a tactile sensor.

7. The system according to claim 6, characterized in that, The tactile sensor includes at least one of the following: an inertial navigation sensor, a suspension height sensor, a wheel speed sensor, a wheel rotation speed sensor, a torque sensor, a steering wheel angle sensor, a brake pedal sensor, and a brake torque sensor.

8. The system according to claim 6, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on sensor data sent by the multiple sensors, including: Based on the sensor data sent by each sensor, the initial road surface recognition result corresponding to each sensor is determined; Based on the initial road surface recognition results corresponding to each of the sensors, the road surface recognition results of the road on which the vehicle is traveling are determined.

9. The system according to claim 8, characterized in that, The initial road surface recognition result includes the original probability corresponding to each candidate road surface type; the road surface recognition result includes the road surface type of the road on which the vehicle travels.

10. The system according to claim 9, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on the initial road surface recognition result corresponding to each of the sensors, including: If the road surface type is consistent in the initial road surface recognition results, the initial road surface recognition results shall be used as the road surface recognition results of the road where the vehicle is traveling.

11. The system according to claim 9, characterized in that, The controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each of the sensors, and further includes: In the case where the road surface types in the initial road surface recognition results are inconsistent, the probability difference between the initial road surface recognition results corresponding to each sensor is determined based on the original probability of the initial road surface recognition results corresponding to each sensor. If the probability difference meets the preset conditions, the initial road surface recognition result corresponding to the original maximum probability is taken as the road surface recognition result of the road where the vehicle is traveling.

12. The system according to claim 11, characterized in that, The preset conditions include: The probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest, and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

13. The system according to claim 9, characterized in that, The controller is configured to determine the road surface recognition result of the road where the vehicle is traveling based on the initial road surface recognition result corresponding to each of the sensors, and further includes: Based on the original probabilities of each candidate road type in the initial road identification results corresponding to each sensor, the comprehensive probability of each candidate road type is determined. The road surface type of the vehicle's travel route is determined based on the combined probability of each candidate road surface type.

14. The system according to claim 13, characterized in that, The controller is configured to determine the comprehensive probability of each candidate road surface type based on the original probabilities of each candidate road surface type in the initial road surface recognition results corresponding to each of the sensors, including: The original probabilities corresponding to the same candidate road surface type in the initial road surface recognition results of each of the sensors are weighted and fused to obtain the comprehensive probability of the candidate road surface type.

15. The system according to claim 14, characterized in that, In the weighted fusion process, the weights corresponding to the visual sensors are determined based on the visibility of the environment in which the vehicle is located.

16. The system according to claim 14, characterized in that, During the weighted fusion, the weight corresponding to the tire noise sensor is determined based on at least one of the noise of the vehicle's environment and the vehicle's motion state.

17. The system according to claim 14, characterized in that, During the weighted fusion, the weights of the tactile sensors are determined based on the motion state of the vehicle.

18. The system according to claim 14, characterized in that, During the weighted fusion, the weights corresponding to each sensor are determined in the following way: Based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined. The weights corresponding to each sensor are determined based on the similarity between the first historical road surface identification results and the second historical road surface identification results of each sensor.

19. The system according to claim 18, characterized in that, The controller is configured to determine the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road on which the vehicle travels, including: The attention scores of the first historical road surface recognition results and the second historical road surface recognition results of each sensor are determined based on the attention mechanism. Based on the attention scores of each sensor, the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor is determined.

20. The system according to claim 19, characterized in that, The controller is configured to determine the similarity between a first historical road surface recognition result and a second historical road surface recognition result of each sensor based on the attention score of each sensor, including: The attention scores of each sensor are normalized to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

21. The system according to claim 10, characterized in that, The controller is also configured to: If the road surface types in the initial road surface recognition results are consistent, before using the initial road surface recognition results as the road surface recognition results of the road where the vehicle is traveling, the initial road surface recognition results corresponding to each sensor are filtered based on the ambient temperature to obtain the filtered initial road surface recognition results corresponding to each sensor.

22. The system according to claim 21, characterized in that, The controller is configured to filter the initial road surface recognition results corresponding to each sensor based on ambient temperature, obtaining the filtered initial road surface recognition results corresponding to each sensor, including: When the ambient temperature is greater than a preset temperature threshold, the initial road surface recognition results containing snow are deleted, and the initial road surface recognition results corresponding to each of the sensors are obtained after filtering.

23. The system according to any one of claims 13 to 22, characterized in that, The controller is configured to determine the road surface type of the road on which the vehicle travels based on the comprehensive probability of each of the candidate road surface types, including: The combined probability of each candidate road surface type is smoothed by using a moving average function, and the smoothed combined probability of each candidate road surface type is determined. The road surface type of the vehicle's travel route is determined based on the combined probability of each candidate road surface type after the smoothing process.

24. The system according to any one of claims 13 to 22, characterized in that, The controller is configured to determine the road surface type of the road on which the vehicle travels based on the comprehensive probability of each of the candidate road surface types, including: The candidate road surface type corresponding to the maximum comprehensive probability is determined as the road surface type of the road on which the vehicle travels.

25. The system according to any one of claims 1 to 24, characterized in that, The tire noise sensor includes tire noise sensors for each wheel in at least one wheel.

26. The system according to claim 25, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on sensor data sent by the multiple sensors, including: Based on the sensor data sent by the tire noise sensors of each wheel, the tire noise characteristics of each wheel are determined. Based on the tire noise characteristics of each wheel, the road surface identification result of the road on which the vehicle is traveling is determined.

27. The system according to claim 26, characterized in that, The controller is configured to determine the road surface identification result of the road surface on which the vehicle is traveling based on the tire noise characteristics of each wheel, including: Based on the tire noise characteristics of each wheel, the road surface recognition result of each wheel is determined; Based on the road surface recognition results of each wheel, the road surface recognition results of the road on which the vehicle travels are determined.

28. The system according to claim 27, characterized in that, The determination of the road surface recognition result for each wheel based on the tire noise characteristics of each wheel includes: The tire noise characteristics of each wheel are fused to obtain the fused tire noise characteristics of each wheel. Based on the tire noise characteristics of each wheel after fusion, the road surface recognition result of each wheel is determined.

29. The system according to claim 28, characterized in that, The process of fusing the tire noise characteristics of each wheel to obtain the fused tire noise characteristics of each wheel includes: The tire noise features of each wheel are spliced ​​together to obtain the fused tire noise features of each wheel.

30. The system according to any one of claims 26-29, characterized in that, The tire noise characteristics include at least one of the following: statistical characteristics, Mel time-frequency characteristics, and wavelet transform time-frequency characteristics.

31. The system according to any one of claims 1 to 30, characterized in that, The controller is further configured to: when the current driving mode and the target driving mode of the vehicle are inconsistent, determine whether to switch from the current driving mode to the target driving mode based on the safety level of the current driving mode and the safety level of the target driving mode.

32. The system according to claim 31, characterized in that, The controller is specifically configured as follows: If the safety level of the current driving mode is lower than the safety level of the target driving mode, the vehicle is determined to switch from the current driving mode to the target driving mode. or, If the safety level of the current driving mode is higher than the safety level of the target driving mode, and the vehicle meets the driving mode switching conditions, then the vehicle is determined to switch from the current driving mode to the target driving mode.

33. The system according to any one of claims 1 to 32, characterized in that, The controller is also configured to determine the target driving mode based on the road surface recognition results of the road where the vehicle is traveling.

34. The system according to claim 33, characterized in that, The vehicle control system also includes an actuator connected to the controller; The controller is also configured to control the actuator to perform the adjustment action corresponding to the target driving mode.

35. The system according to claim 34, characterized in that, The actuator includes at least one of the following: electronic differential, power subsystem, braking subsystem, steering subsystem, and active suspension subsystem.

36. The system according to any one of claims 1 to 35, characterized in that, The controller includes a smart driving domain controller; The intelligent driving domain controller is configured to: receive sensor data sent by the sensors; and determine the road surface recognition result of the road where the vehicle is traveling based on the sensor data sent by the various sensors.

37. The system according to claim 36, characterized in that, The controller also includes a vehicle controller; The intelligent driving domain controller is also configured to send the road surface recognition results of the road where the vehicle is traveling to the vehicle controller.

38. The system according to claim 36, characterized in that, The controller also includes a vehicle controller; The intelligent driving domain controller is also configured to: determine the target driving mode based on the road surface recognition results of the road where the vehicle is traveling; and send the target driving mode to the vehicle controller. The vehicle controller is configured to control the actuator to perform the adjustment action corresponding to the target driving mode.

39. The system according to any one of claims 1-38, characterized in that, The vehicle control system further includes: a human-machine interface device connected to the controller; The human-computer interaction device is configured to: generate a road surface recognition command and send the road surface recognition command to the controller; The controller is configured to: in response to the road surface recognition command, determine the road surface recognition result of the road surface on which the vehicle is traveling based on sensor data sent by the multiple sensors.

40. The system according to claim 39, characterized in that, The road surface recognition command is triggered in at least one of the following ways: Receives touch operations from the user on the display interface of the in-vehicle display device; Received the user's trigger operation on the road surface recognition button; The system receives audio signals carried by the user for road surface recognition.

41. The system according to claim 39, characterized in that, The controller is also configured to send the road surface recognition result to the human-computer interaction device; The human-computer interaction device is also configured to receive and display the road surface recognition results sent by the controller.

42. The system according to claim 39, characterized in that, The controller is also configured to send an information display instruction carrying prompt information to the human-machine interaction device when the current driving mode and the target driving mode of the vehicle are inconsistent. The prompt information includes: the road surface recognition result and the target driving mode; The human-computer interaction device is further configured to: receive and parse the information display instruction to obtain the prompt information; and display the prompt information.

43. The system according to claim 42, characterized in that, The prompt message is displayed in at least one of the following ways: The prompt information is displayed through the display interface of the in-vehicle display device; The prompt message is output via the vehicle's audio system.

44. The system according to claim 41, characterized in that, The human-computer interaction device is also configured to generate a confirmation command for the target driving mode; The controller is also configured to control the vehicle to drive based on the target driving mode in response to a confirmation command for the target driving mode.

45. The system according to claim 44, characterized in that, The confirmation command for the target driving mode is triggered in at least one of the following ways: Receives a driving mode confirmation from the user on the display interface of the in-vehicle display device; Received the user's trigger operation on the driving mode confirmation button; Received an audio signal from the user confirming the target driving mode.

46. ​​The system according to claim 6, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on sensor data sent by the multiple sensors, including: The sensor data sent by the vision sensor is segmented to obtain multiple regional sub-images; the sensor data includes: a road surface image of the road where the vehicle is currently traveling; The categories of the multiple regional sub-images are filtered to determine the regional sub-images of the category of road surface; the category of the regional sub-images includes at least one of the following: road surface, obstacle, lane line; Based on the sub-image of the area categorized as road surface, the road surface recognition result of the road currently being traveled by the vehicle is determined.

47. The system according to claim 46, characterized in that, The controller is configured to segment the road surface image of the road currently being traveled by the vehicle to obtain multiple regional sub-images, including: The road surface image is semantically segmented based on a neural network model to determine the category of each pixel in the road surface image; Based on the category of each pixel, the road surface image is divided into multiple sub-images of the region.

48. The system according to claim 6, characterized in that, The controller is configured to determine the road surface recognition result of the road surface where the vehicle is traveling based on sensor data sent by the multiple sensors, including: Feature extraction is performed on the sensor data sent by the tactile sensor to obtain the motion characteristics of the vehicle; Based on the motion characteristics, the road surface characteristics of the road surface on which the vehicle is currently traveling are determined; Based on the road surface features, the road surface recognition result of the road currently being traveled by the vehicle is determined.

49. The system according to claim 48, characterized in that, The motion characteristics include at least one of the following: four-wheel load, vehicle speed, gradient, rolling resistance, slip ratio, adhesion coefficient, and road surface smoothness.

50. A vehicle control method, characterized in that, Applied to the vehicle control system according to any one of claims 1-38, the method comprises: Acquire sensor data from multiple sensors; the multiple sensors include at least: a tire noise sensor; Based on the sensor data sent by the various sensors, the road surface identification result of the road on which the vehicle is traveling is determined.

51. The method according to claim 50, characterized in that, The determination of the road surface recognition result based on the data from the multiple sensors includes: If the road surface type is consistent in the initial road surface recognition results, the initial road surface recognition results shall be used as the road surface recognition results of the road where the vehicle is traveling.

52. The method according to claim 50, characterized in that, The method of determining the road surface recognition result of the road where the vehicle is traveling based on the data from the multiple sensors also includes: In the case where the road surface types in the initial road surface recognition results are inconsistent, the probability difference between the initial road surface recognition results corresponding to each sensor is determined based on the original probability of the initial road surface recognition results corresponding to each sensor. If the probability difference meets the preset conditions, the initial road surface recognition result corresponding to the maximum confidence value is taken as the road surface recognition result of the road where the vehicle is traveling.

53. The method according to claim 52, characterized in that, The preset conditions include: The probability difference between the original probability of the first initial road surface recognition result and the original probability of the second initial road surface recognition result is greater than a preset probability threshold; the original probability of the first initial road surface recognition result is the highest, and the second initial road surface recognition result is an initial road surface recognition result other than the first initial road surface recognition result.

54. The method according to claim 50, characterized in that, The step of determining the road surface recognition result of the road surface based on the initial road surface recognition results corresponding to each of the sensors further includes: Based on the original probabilities of each candidate road type in the initial road identification results corresponding to each sensor, the comprehensive probability of each candidate road type is determined. The road surface type of the vehicle's travel route is determined based on the combined probability of each candidate road surface type.

55. The method according to claim 54, characterized in that, The determination of the comprehensive probability of each candidate road surface type based on the original probabilities of each candidate road surface type in the initial road surface recognition results corresponding to each of the sensors includes: The original probabilities corresponding to the same candidate road surface type in the initial road surface recognition results of each of the sensors are weighted and fused to obtain the comprehensive probability of the candidate road surface type.

56. The method according to claim 55, characterized in that, During the weighted fusion, the weights corresponding to each sensor are determined in the following way: Based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the road where the vehicle travels, the similarity between the first historical road surface recognition results and the second historical road surface recognition results of each sensor is determined. The weights corresponding to each sensor are determined based on the similarity between the first historical road surface identification results and the second historical road surface identification results of each sensor.

57. The method according to claim 56, characterized in that, The determination of the similarity between the first historical road surface recognition results and the second historical road surface recognition results of the roads traveled by the vehicle, based on the first historical road surface recognition results of each sensor and the second historical road surface recognition results of the roads traveled by the vehicle, includes: The attention scores of the first historical road surface recognition results and the second historical road surface recognition results of each sensor are determined based on the attention mechanism. Based on the attention scores of each sensor, the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor is determined.

58. The method according to claim 57, characterized in that, The determination of the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor based on the attention score of each sensor includes: The attention scores of each sensor are normalized to determine the similarity between the first historical road surface recognition result and the second historical road surface recognition result of each sensor.

59. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 50-58.

60. A computer-readable storage medium, characterized in that, When the computer-executable instructions stored in the computer-readable storage medium are executed by the processor of the device, the device is capable of performing the method as described in any one of claims 50-58.

61. A vehicle, characterized in that, include: The vehicle control system as claimed in any one of claims 1-49, or the electronic device as claimed in claim 59, or the computer-readable storage medium as claimed in claim 60.

62. A computer program product, the computer program product comprising computer instructions, characterized in that, When the computer instructions are executed on the processor of the device, the device is enabled to perform the method as described in any one of claims 50-58.

Images