Vehicle control method, server, vehicle, and storage medium

Abstract

This invention relates to the field of intelligent driving technology, and provides a vehicle control method, a server, a vehicle, and a storage medium. The method includes: acquiring first sensor data of the road surface on which the vehicle is traveling; obtaining a first judgment result regarding the existence of an anomaly on the road surface based on the first sensor data; if the first judgment result indicates an anomaly on the road surface, acquiring second sensor data of the road surface; obtaining a second judgment result regarding the existence of an anomaly on the road surface based on the second sensor data; and if the second judgment result indicates an anomaly on the road surface, controlling the vehicle to execute a driving strategy for the abnormal road surface. This method processes point cloud and image data simultaneously only when an anomaly is initially determined to exist on the road surface; under other conditions, it processes only point cloud or image data, effectively reducing computational load while ensuring the accuracy of abnormal road surface identification.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more specifically to vehicle control methods, servers, vehicles, and storage media. Background Technology

[0002] Passenger cars are one of the main modes of transportation for people today. Despite continuous improvements in road infrastructure in recent years, the rapid increase in vehicles on the road has led to more complex road conditions and traffic environments, increasing the incidence of traffic accidents. To provide drivers with a better driving experience and reduce the accident rate, passenger cars are equipped with various sensors to acquire real-time information about the internal and external environment, assisting drivers in controlling the vehicle or alerting them to potential hazards ahead. Current technology uses sensors installed at the front of the vehicle to acquire real-time data on the road ahead, identifying road conditions such as continuous speed bumps, gravel, mud, potholes, and snow in advance, allowing the active suspension to adjust accordingly to ensure smooth operation, vehicle passability, comfort, and safety.

[0003] However, existing road surface detection methods have the following problems: If road surface information obtained by sensing devices such as LiDAR and cameras is to be analyzed simultaneously, point cloud data obtained by LiDAR and image data obtained by sensing devices such as cameras must be processed at the same time under all working conditions. The onboard computing power is difficult to meet the computing requirements, let alone ensure the robustness of the results. Analyzing road surface information from a single sensing device will introduce computational bias. Although the amount of computation is reduced, the accuracy of abnormal road surface identification cannot be guaranteed, nor can the safety and comfort requirements of vehicle passage be met. Summary of the Invention

[0004] The technical problem to be solved by the present invention is the deviation problem caused by using a single sensor to acquire road surface data and the problem that the on-board computing power is insufficient to meet the requirements for processing point cloud data and image data at the same time.

[0005] To achieve the above objectives, the present invention provides a road surface detection method, the steps of which include:

[0006] Acquire first sensor data of the road surface on which the vehicle is traveling;

[0007] Based on the data from the first sensor, a first judgment result is obtained regarding whether there is an anomaly on the road surface.

[0008] If the first judgment result indicates that there is an anomaly on the driving road surface, the second sensor data of the driving road surface shall be acquired;

[0009] Based on the data from the second sensor, a second judgment result is obtained regarding whether there is an anomaly on the road surface.

[0010] Based on the vehicle's driving mode, an anomaly is determined to exist on the road surface by one of the following conditions:

[0011] Both the first and second judgment results indicate that the road surface is abnormal; or, either the first or second judgment result indicates that the road surface is abnormal.

[0012] In this process, different types of sensors are used to collect data from the first sensor and the second sensor.

[0013] Optionally, the first sensor data and the second sensor data are collected by the vehicle-mounted sensors and transmitted to the cloud server; after obtaining the first sensor data and the second sensor data, the cloud server performs the following steps:

[0014] Based on the data from the first sensor, a first judgment result is obtained regarding whether there is an anomaly on the road surface; simultaneously, based on the data from the second sensor, a second judgment result is obtained regarding whether there is an anomaly on the road surface.

[0015] Based on the vehicle's driving mode, an abnormality is determined to exist on the road surface by one of the following conditions: both the first determination result and the second determination result indicate that the road surface is abnormal; or, either the first determination result or the second determination result indicates that the road surface is abnormal.

[0016] Preferably, the first sensor data is one of road surface point cloud data and road surface image data; the second sensor data is the other of road surface point cloud data and road surface image data.

[0017] Preferably, the road surface point cloud data is acquired using one of lidar, millimeter-wave radar, or ultrasonic radar; and the road surface image data is acquired using one of a monocular camera, binocular camera, tricular camera, or surround-view camera.

[0018] Preferably, when the first sensor data or the second sensor data is the road surface point cloud data, the step of obtaining a first judgment result on whether there is an anomaly on the driving road surface based on the first sensor data, or the step of obtaining a second judgment result on whether there is an anomaly on the driving road surface based on the second sensor data, includes the following steps:

[0019] Obtain the three-dimensional coordinates of all road points in the road point cloud data;

[0020] Calculate the plane S of the driving road surface;

[0021] Traverse all road surface points; if the distance between a road surface point and the plane S exceeds a first preset threshold, then the road surface point is an anomaly.

[0022] The cumulative number of abnormal points is calculated. If the number of abnormal points is greater than a second preset threshold, it is determined that there is an abnormality in the road surface; otherwise, there is no abnormality.

[0023] Preferably, the method for calculating the plane S of the driving road surface includes the following steps:

[0024] Obtain the normal vector of the chassis plane of the vehicle, and use the normal vector of the chassis plane of the vehicle as the normal vector (a, b, c) of the plane S;

[0025] Calculate the mean point A0(x0,y0,z0) of all road surface points within the road surface point cloud data; and

[0026] According to the point method formula for a plane: a(x-x0)+b(y-y0)+c(z-z0)=0, the equation of the plane S is obtained as: ax+by+cz+d=0, where x, y, z are the coordinates of a point on the plane S, and a, b, c, d are the parameters of the equation of the plane S.

[0027] Preferably, when the first sensor data or the second sensor data is the road surface image data, the step of obtaining a first judgment result regarding whether there is an anomaly on the road surface based on the first sensor data, or the step of obtaining a second judgment result regarding whether there is an anomaly on the road surface based on the second sensor data, includes:

[0028] The road surface image data is input into the trained deep learning model to obtain the judgment result of whether there is an anomaly on the driving road surface output by the deep learning model.

[0029] Compared with existing technologies, the road surface detection method of the present invention has the following advantages:

[0030] (1) By collecting road image data through sensing devices such as the front camera of the vehicle and collecting road point cloud data through LiDAR, it is possible to comprehensively obtain environmental information in front of the vehicle and make more accurate judgments on abnormal road conditions.

[0031] (2) If the road surface is initially determined to be abnormal by using either point cloud data or image data, the other type of data can be added for a more detailed secondary judgment to further determine whether the road surface is abnormal. This can reduce the amount of calculation and reduce the computational pressure.

[0032] (3) Based on the initial determination of whether the road surface is abnormal, the deep learning model can further effectively identify road surface abnormalities and ensure the accuracy of abnormal road surface identification.

[0033] Another aspect of the present invention provides a vehicle control method, which performs the following steps when the vehicle is traveling on an abnormal road surface:

[0034] The abnormal road surface is modeled in three dimensions to obtain a three-dimensional model of the abnormal road surface.

[0035] Based on the three-dimensional model of the abnormal road surface, obtain the driving parameters of the vehicle passing through the abnormal road surface;

[0036] Based on the aforementioned driving parameters, kinematic model, and vehicle dynamics model, a vehicle suspension adjustment scheme is obtained; and

[0037] Control the vehicle to execute the vehicle suspension adjustment scheme.

[0038] Preferably, the step of performing three-dimensional modeling of the abnormal road surface to obtain a three-dimensional model of the abnormal road surface includes the following steps:

[0039] Based on the image data, calculate the image data feature points and their three-dimensional coordinates;

[0040] Based on the road surface point cloud data, calculate the feature points of the road surface point cloud data and their three-dimensional coordinates;

[0041] The image data feature points and the road surface point cloud data feature points are matched, and invalid points in the image data feature points and the road surface point cloud data are excluded to obtain the matching points and their three-dimensional coordinates.

[0042] By merging the matching points and their three-dimensional coordinates, the three-dimensional model of the abnormal road surface is obtained.

[0043] Preferably, the step of simulating the driving parameters of a vehicle passing through the abnormal road surface based on the three-dimensional model of the abnormal road surface includes the following steps:

[0044] The abnormal road surface 3D model, vehicle driving data, and obstacle distance and angle information are acquired and input into the simulation tool to simulate the vehicle's driving parameters when passing through the abnormal road surface.

[0045] Compared with existing technologies, the vehicle control method described above has the following advantages:

[0046] (1) By collecting road information through the vehicle's front camera and lidar, it is possible to obtain comprehensive information about the environment in front of the vehicle.

[0047] (2) Use the point cloud data of lidar to initially determine whether the road surface is abnormal. If the abnormality is initially determined, use the deep learning model to identify the image captured by the camera to further determine whether the road surface is abnormal. This can reduce the amount of computation for processing point cloud data and image data at the same time and reduce the computational pressure.

[0048] (3) Based on the initial determination of whether the road surface is abnormal, the deep learning model can further effectively identify road surface abnormalities and ensure the accuracy of abnormal road surface identification.

[0049] (4) Extract feature points and three-dimensional coordinates from image data and feature points and three-dimensional coordinates from road surface point cloud data. Then match the feature points obtained from the two methods and eliminate invalid points to obtain a more accurate three-dimensional reconstruction model.

[0050] Another object of the present invention is to provide a server to solve the problem of bias caused by using a single sensor to acquire road surface data and / or the problem of insufficient computing power when relying solely on the local computing power of the vehicle to process point cloud data and image data at the same time.

[0051] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0052] A server configured to execute the road surface detection method and / or vehicle control method described above.

[0053] The server can be a cloud server. The vehicle-mounted equipment uploads road and driving data to the cloud server via 4G / 5G technology. The cloud server then executes any of the vehicle control methods described above and sends the control program / instructions to the vehicle for execution via 4G / 5G technology.

[0054] Compared to existing technologies, the server, in addition to having the technical advantages of the vehicle control method described above, can also better allocate on-board computing power and achieve better autonomous management of the entire vehicle.

[0055] Another object of the present invention is to provide a vehicle that solves the problem of bias caused by using a single sensor to acquire road surface data and / or the problem of insufficient computing power when relying solely on the local computing power of the vehicle to process point cloud data and image data at the same time.

[0056] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0057] A vehicle configured to perform road surface detection methods and / or vehicle control methods as described above.

[0058] The vehicle and the vehicle control method described above have the same technical advantages over existing technologies, which will not be repeated here.

[0059] Another objective of this invention is to provide a machine-readable storage medium to address the bias problem caused by using a single sensor to acquire road surface data and / or the problem of insufficient computing power when relying solely on the local computing power of a vehicle to process point cloud data and image data simultaneously.

[0060] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0061] A machine-readable storage medium storing instructions for causing a machine to perform any of the road surface detection methods and / or vehicle control methods described above.

[0062] Compared to existing technologies, the system has the same technical advantages as the vehicles and servers mentioned above, which will not be repeated here.

[0063] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0064] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0065] Figure 1A and Figure 1B This is a flowchart illustrating an embodiment of the vehicle control method;

[0066] Figure 2 This is a schematic diagram of the road surface anomaly detection process;

[0067] Figure 3 This is a schematic diagram of the road surface anomaly classification and identification process;

[0068] Figure 4 This is a schematic diagram of the 3D reconstruction process;

[0069] Figure 5 This is a schematic diagram of the overall process of a vehicle control method based on 5G communication technology; and

[0070] Figure 6 This is a schematic diagram of the cloud server's computing process. Detailed Implementation

[0071] In this invention, unless otherwise stated, the technical terms used generally have the following meanings:

[0072] Automotive dynamics simulation: Automotive system dynamics is a discipline that studies all aspects of the motion of automotive systems. It is the foundation of automotive optimization design and control. Through simulation analysis, the optimal matching of automotive structural parameters and dynamic characteristics can be achieved, enabling the vehicle to achieve optimal performance.

[0073] Cloud servers: Elastic Compute Service (ECS) is a simple, efficient, secure, reliable computing service with elastically scalable processing power. Its management is simpler and more efficient than that of physical servers. Users can quickly create or release any number of cloud servers without having to purchase hardware in advance.

[0074] 3D Reconstruction: 3D reconstruction refers to establishing a mathematical model of a 3D object that is suitable for computer representation and processing. It is the basis for processing, manipulating and analyzing the properties of the object in a computer environment, and it is also a key technology for establishing virtual reality that expresses the objective world in a computer.

[0075] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0076] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0077] The following is a detailed description of a specific embodiment of the pavement detection method of the present invention, with reference to the appendix. Figure 1A As shown.

[0078] Step S11: Acquire first sensing data of the road surface where the vehicle is traveling. In this embodiment, this specifically involves acquiring point cloud data of the road surface in the direction of travel ahead of the vehicle. In a specific application scenario of this application, the point cloud data can be obtained by a LiDAR installed on the vehicle, or it can be point cloud data acquired by a camera or other sensing devices.

[0079] Step S12: Obtain a first judgment result regarding whether there is an anomaly on the road surface based on the first sensor data. Specifically, in this embodiment, based on the road surface point cloud data, determine whether there is an anomaly on the road surface ahead of the vehicle. If the result indicates no anomaly, continue detecting the road surface ahead, and the vehicle continues driving normally; if the result indicates an anomaly, further identification of the road surface anomaly is required.

[0080] This method utilizes the principle that point cloud points are coplanar when a vehicle is traveling on a flat road, but not on an uneven road surface due to obstacles, to determine if there are any road surface anomalies. A specific implementation method for determining whether a road surface is abnormal can be: acquiring road surface point cloud data, which includes the three-dimensional coordinates of the points; calculating the plane S equation of the road surface based on the coordinates of the road surface point cloud; traversing all road surface points and calculating the deviation distance of each point from plane S; if the distance from a point to the plane is greater than a given distance threshold range, the point is determined to be an anomaly; finally, the sum of all anomaly points is calculated; if the number of anomaly points is greater than a given threshold, the road surface is considered abnormal; if it is less than the given threshold, the road surface is considered normal. Other methods known to those skilled in the art can also be used to determine whether a road surface is abnormal. In this embodiment, refer to... Figure 2 The process shown is explained in detail below.

[0081] Step S13: If the first judgment result indicates that there is an anomaly on the road surface, acquire second sensor data of the road surface. In this embodiment, specifically, if there is an anomaly on the road surface, acquire image data of the direction of travel in front of the vehicle. The image data is preferably obtained through a binocular camera installed on the vehicle. If the first judgment result indicates that there is no anomaly on the road surface, depending on the driving mode, normal driving can continue, and step S11 can be performed to identify road surface anomalies based on the first sensor.

[0082] Step S14: Obtain a second judgment result regarding whether there are abnormalities on the road surface based on the second sensor data. Specifically, in this embodiment, this involves identifying road surface anomalies based on the image data and a deep learning model. The image data captured by the binocular camera is input into a pre-trained deep learning model to identify road surface anomaly categories. This deep learning model has the ability to classify road conditions such as potholes, continuous speed bumps, gravel, and obstacles.

[0083] The deep learning model includes convolutional networks and fully connected layers. A method for identifying road surface anomalies can be: acquiring an image of the abnormal road surface, using a convolutional neural network to extract features from the image to obtain feature vectors, inputting the feature vectors into a fully connected layer, classifying the feature vectors, and outputting a numerical value representing a certain type of road surface anomaly; different output values ​​indicate different road surface anomaly categories. Other methods known to those skilled in the art can also be used to identify road surface anomalies. In this embodiment, reference can be made to... Figure 3 The process shown is explained in detail below.

[0084] Step S15: Determine whether there is an anomaly on the road surface based on the first and second judgment results. Specifically, in this embodiment: if both the first and second judgment results indicate an anomaly on the road surface, then execute the vehicle control method. The execution content of the vehicle control method will be further described below; see appendix for details. Figure 1B And related content.

[0085] By implementing the above road surface detection method, the following beneficial effects can be obtained compared with the existing technology: (1) Point cloud data is used to initially determine whether the road surface is abnormal. If the abnormality is initially determined, the deep learning model is used to identify the image data in the direction of the vehicle's travel to further determine whether the road surface is abnormal. This can reduce the amount of computation of processing point cloud data and image data simultaneously and reduce the computational pressure; (2) The combination of road surface point cloud data and image data can comprehensively obtain the environmental information in front of the vehicle and accurately identify road surface abnormalities; (3) The deep learning model can effectively identify the types of road surface abnormalities.

[0086] It is understood that the initial judgment of road anomalies is not limited to road surface point cloud data obtained by LiDAR; it can also be road surface point cloud data obtained by other sensing devices, or image data obtained by other sensing devices. When an anomaly is initially determined to exist on the road surface, a secondary judgment can be made by differentiating the sensing devices and data types used in the initial judgment, thereby more accurately identifying the abnormal road surface. The above implementation steps are only one method used in this embodiment and are not intended to limit the invention. The first sensor data and the second sensor data mentioned in claim 1 do not represent a strict order of acquiring road surface point cloud data and road surface image data in actual implementation.

[0087] Generally speaking, road point cloud data is obtained through various radar sensors, containing road point coordinate information. The data accuracy is high and it is less affected by weather. Road image data, on the other hand, is obtained through various camera sensors. Although the data accuracy is lower, it contains features such as image color and texture. The data volume is small and the computing power requirement is low.

[0088] In the specific implementation process, technicians can choose to process the road point cloud data first, or process the road image data first, or even process both simultaneously, depending on factors such as vehicle computing power configuration, road surface and weather conditions, driving mode, and cloud server configuration.

[0089] Furthermore, when higher driving safety requirements are in place, an anomaly is considered to exist and further vehicle control measures are implemented if the processing results of either the road surface point cloud data or the road surface image data indicate an anomaly in the road surface. Conversely, when more computing power needs to be allocated to entertainment or other aspects, and the road surface condition is relatively good, the vehicle is allowed to execute an alternative road surface detection strategy. That is, an anomaly is only considered to exist and further vehicle control measures are implemented if the processing results of both the road surface point cloud data and the road surface image data indicate an anomaly in the road surface.

[0090] When the road surface detection method indicates an anomaly on the road surface, the vehicle control method described in this application can be executed to enable the vehicle to travel smoothly and safely over the abnormal road surface. The vehicle control method described in this application is based on the premise that an anomaly exists on the road surface; the judgment steps are described in the above embodiments and will not be repeated here.

[0091] After the road surface detection method implementation steps in the above embodiments are completed, the vehicle operating environment is further simulated using modeling and simulation software. The vehicle suspension adjustment commands of the vehicle are then calculated using the kinematic and vehicle dynamics models included in the modeling and simulation software. The steps are as follows: Figure 1B As shown, after step S15 in the aforementioned embodiment, the following steps are further performed:

[0092] Step S16: After identifying road surface anomalies, further perform 3D modeling of the abnormal road surface to obtain a 3D model of the abnormal road surface. The 3D modeling method can be as follows: acquire road surface point cloud data of the road surface in front of the vehicle obtained by LiDAR; calculate the 3D coordinates of LiDAR feature points from the road surface point cloud data using triangulation; detect, correct offset points, and eliminate invalid points using the LiDAR road surface point cloud to obtain the 3D coordinates of matching points; and convert the 3D coordinates of the matching points to obtain a 3D reconstruction model. Other methods known to those skilled in the art can also be used for 3D modeling. In this embodiment, the 3D modeling process for abnormal road surfaces can be referred to... Figure 4 As shown, the specific implementation method will also be introduced later.

[0093] Understandably, the 3D model of abnormal road surface includes 3D information of road obstacles, which is used to calculate the real-time distance and orientation of obstacles to the vehicle during driving, as well as parameters such as the height, attitude, and shape of obstacles.

[0094] After completing the 3D model of the abnormal road surface, the model can be shared with nearby vehicles through communication methods, helping them to quickly obtain information about the road abnormality and pass through safely and orderly.

[0095] Step S17: Based on the 3D model of the abnormal road surface, obtain the driving parameters of the vehicle passing through the abnormal road surface. After obtaining the 3D model of the abnormal road surface, further obtain obstacle distance, angle, and vehicle speed data to simulate the relevant parameters of the vehicle passing through the abnormal road surface.

[0096] Step S18: Calculate the vehicle suspension adjustment scheme for the vehicle based on the driving parameters and kinematic model of the vehicle passing through the abnormal road surface and the vehicle dynamics model.

[0097] Step S19: Control the vehicle to execute the vehicle suspension adjustment scheme.

[0098] Preferably, the relevant parameters of the vehicle passing through abnormal road surfaces include: vehicle speed, steering angle, suspension vertical force, camber angle, toe angle, vertical force, suspension geometry data, center of gravity sideslip angle, and path curvature.

[0099] Preferably, the method for obtaining the distance and angle of obstacles can be a calculation method based on three-dimensional modeling of abnormal road surfaces: Binocular stereo vision is used to fuse LiDAR data to reconstruct the obstacle in three dimensions. The binocular stereo vision reconstruction method simulates the stereo imaging principle of human eyes. Left and right cameras simultaneously capture images of an obstacle in the scene. Through trigonometric relationships and the principle of parallax, the coordinate information of points on the obstacle's surface from the same viewpoint of the left and right cameras is obtained, thereby constructing the obstacle's position and shape. LiDAR three-dimensional reconstruction involves emitting a detection signal (laser beam) towards the obstacle, then comparing the received signal reflected back from the obstacle (target echo) with the emitted signal to obtain relevant information about the obstacle, such as the obstacle's distance and orientation relative to the vehicle, as well as parameters such as the obstacle's height, attitude, and shape.

[0100] Preferably, vehicle driving data includes: real-time vehicle location, speed, acceleration, gear information, shift mode, vehicle driving trajectory, braking signals, and other parameters related to vehicle operation.

[0101] By implementing the above steps, abnormal conditions on the road ahead of the vehicle can be predicted and processed in advance, and converted into driving instructions to ensure safe and stable driving of the vehicle.

[0102] for Figure 1A and Figure 1B The methods for determining whether there are abnormalities on the road ahead, identifying the type of abnormal road surface, and 3D modeling in the embodiments described below will be introduced in detail below.

[0103] The procedure for determining whether there are any abnormalities on the road surface ahead of the vehicle can be found in the following reference. Figure 2As shown, when a vehicle is traveling on a flat road, the point cloud points are coplanar. However, when there are obstacles or the road surface is uneven, a certain number of points in the point cloud are not coplanar, allowing for the determination of whether there are road anomalies. The specific steps include:

[0104] S201: Read road surface point cloud data, which includes the three-dimensional coordinates of road surface points;

[0105] S202: Calculate the equation of plane S of the driving road surface. The normal vector of the vehicle chassis plane is obtained through sensors of the chassis inertial navigation system. This normal vector is used as the normal vector (a, b, c) of plane S. Then, the mean point A0(x0, y0, z0) of the road surface point cloud is calculated. According to the point normal formula for a plane: a(x-x0) + b(y-y0) + c(z-z0) = 0, the equation of plane S is obtained: ax + by + cz + d = 0, where x, y, z are the coordinates of a point on plane S, and a, b, c, d are the parameters of the equation of plane S obtained by the aforementioned method.

[0106] It is understandable that, during vehicle operation, using the normal vector of the vehicle chassis as the normal vector of the current road surface, and the mean point of the road surface point cloud as a point on the current road surface, is a method that closely approximates reality. Obtaining the normal vector of the vehicle chassis plane through sensors in the chassis inertial navigation system is also a simulation method. As the vehicle travels, the road surface detection method is continuously executed according to a preset cycle, and the calculated plane S of the driving road surface is constantly corrected and simulates the actual road conditions. In actual implementation, those skilled in the art can also adopt other similar methods based on the ideas of this application, and such methods, without departing from the spirit of this invention, should also be within the scope of protection of this invention.

[0107] S203: Traverse the point cloud of road surfaces and calculate the distance from each point to plane S. Calculate using the point-to-plane equation formula:

[0108]

[0109] Calculate the distances from the road surface point cloud points to the plane S, and obtain the distance set {d1, d2, d3, ..., d...} i According to the distance set, when the distance from a point to the plane exceeds the first preset threshold, the point is determined to be an anomaly. Finally, the distances of all anomalies are summed.

[0110] It is understandable that the road surface is uneven. The distance between a point in the point cloud data and the plane S representing the road surface reflects the vertical deviation of the point from the plane S. These values ​​are all positive. Therefore, the first preset threshold is a positive value. The specific value is an empirical value set based on factors such as the chassis height of the vehicle model.

[0111] S204: When the number of abnormal points is greater than the second preset threshold, the road surface is considered to be abnormal; when it is less than the given threshold, the road surface is considered to be normal.

[0112] It is understandable that, depending on factors such as vehicle type, road conditions, and driving mode, different empirical values ​​may be assigned to the second preset threshold during actual driving.

[0113] The implementation process for identifying road surface anomalies can be found in the following reference. Figure 3 As shown, the specific steps include:

[0114] S301: Acquire images of abnormal road surfaces ahead of the vehicle. These images can be obtained using a binocular camera.

[0115] S302: Use the convolution algorithm to extract features from the image to obtain a feature vector;

[0116] S303: The extracted feature vectors are classified using a convolutional neural network, mapping each type of feature vector to a corresponding label vector. Specifically, the feature vectors are input into a fully connected layer for classification, and a numerical value is output, which represents a certain type of road surface anomaly. Different output values ​​indicate different road surface anomaly categories.

[0117] In particular, the deep learning model is a pre-trained deep learning model that has the ability to classify road conditions such as potholes, continuous speed bumps, gravel, and obstacles.

[0118] After identifying the road surface anomalies, a 3D model of the abnormal road surface is needed to obtain a 3D model of the abnormal road surface. In this embodiment, binocular stereo vision is used to reconstruct the road surface by fusing LiDAR data. The binocular stereo vision reconstruction method simulates the stereo imaging principle of human eyes. The left and right cameras simultaneously capture images of an object in the scene. Through trigonometric relationships and the principle of parallax, the coordinate information of the object's surface points from the same viewpoint of the left and right cameras is obtained, thereby constructing the object's position and shape. The LiDAR emits a detection signal (laser beam) towards the target, and then compares the received signal reflected back from the target (target echo) with the emitted signal to obtain relevant information about the target, such as target distance, orientation, height, attitude, and even shape parameters. By fusing the LiDAR measurement parameters, problems such as image matching errors and positioning errors in the binocular stereo vision 3D reconstruction are corrected, resulting in a more accurate 3D reconstruction model of the abnormal road surface.

[0119] The process for 3D modeling abnormal road surfaces can be found in the following reference. Figure 4 As shown, the specific implementation steps include:

[0120] S401: The binocular camera captures images of the road surface in front of the vehicle using both the left and right cameras. The images of the road surface in front of the vehicle, captured by the binocular camera, are obtained as left and right view images.

[0121] S402: Simultaneously extract the features of the left eye image and the feature points on the left eye camera image. Match the feature points of the left eye image and the right eye image according to the SGBM algorithm, eliminate the points that cannot be matched, and obtain the feature points of the binocular camera.

[0122] S403: Calculate the three-dimensional coordinates of the feature points of the binocular camera based on triangulation and the parallax principle. The three-dimensional coordinates of the feature points of the binocular camera are real-world coordinates with the vehicle as the reference center.

[0123] S404: Acquire road surface point cloud data of the road surface in front of the vehicle obtained by LiDAR; calculate the three-dimensional coordinates of the LiDAR feature points based on the LiDAR road surface point cloud data using triangulation. Similarly, the three-dimensional coordinates of the LiDAR feature points are real-world coordinates with the vehicle as the reference center.

[0124] S405: Match the three-dimensional coordinates of the feature points of the lidar and the feature points of the binocular camera, and exclude invalid points of the image data feature points and the road surface point cloud data to obtain the matching points and their three-dimensional coordinates;

[0125] Understandably, feature points whose 3D coordinates obtained from LiDAR data and binocular camera data are within the error range can be considered as matching feature points, while unmatched points are excluded as invalid points.

[0126] S406: Merge the matching points and their three-dimensional coordinates to obtain a three-dimensional reconstruction model.

[0127] By implementing the above vehicle control method, the following beneficial effects can be obtained compared with the existing technology: (1) The point cloud data of the lidar is used to initially determine whether the road surface is abnormal. If the abnormality is initially determined, the deep learning model is used to identify the image captured by the camera to further determine whether the road surface is abnormal. This can reduce the amount of computation for processing point cloud data and image data at the same time and reduce the computational pressure; (2) The deviation algorithm between the point cloud data and the driving road surface plane is used to initially determine whether the road surface is abnormal. This can effectively identify abnormal road surfaces and ensure the accuracy of abnormal road surface identification; (3) The combination of road surface point cloud data and image data can comprehensively obtain the environmental information in front of the driving vehicle; (4) The deep learning model can effectively identify the road surface abnormality category; (5) By matching the features of the left and right binocular images of the binocular camera, and then by matching the feature points obtained by the lidar road surface point cloud data with the feature points of the binocular camera, and by detecting, correcting the offset points and eliminating invalid points through the lidar road surface point cloud, a more accurate three-dimensional reconstruction model can be obtained.

[0128] In the above embodiment, the point cloud data from the LiDAR is used to initially determine whether the road surface is abnormal. Then, a deep learning model is used to identify images captured by a camera to further determine whether the road surface is abnormal. It is understood that this embodiment is only a feasible preferred implementation method and is not actually limited to this order, nor is it limited to using LiDAR to acquire point cloud data and using a camera to acquire image data. Other sensors can be used as long as road surface point cloud data and image data can be acquired. Those skilled in the art can acquire and analyze image data first, then acquire and analyze road surface point cloud data, or even acquire and analyze image data and road surface point cloud data simultaneously. Anything that conforms to the spirit of this invention should be within the scope of protection.

[0129] For situations where the vehicle's local computing power is insufficient, another embodiment of the present invention combines 5G communication technology to connect the vehicle to a cloud server. The implemented vehicle control method can be found in [reference needed]. Figure 5 As shown, the specific implementation steps are as follows:

[0130] S501: Road information collection components and systems can be implemented by moving vehicles;

[0131] S502: The collected road information data and necessary vehicle driving parameters are packaged and uploaded to the cloud server via 5G communication technology;

[0132] S503: The cloud server obtains vehicle suspension adjustment commands by performing road anomaly checks, abnormal road surface 3D modeling, simulation and modeling steps;

[0133] S504: Vehicle suspension adjustment commands are transmitted to the vehicle via 5G communication technology;

[0134] Ultimately, the driving instructions are executed by the vehicle itself.

[0135] The implementation methods for road information collection, road anomaly detection, abnormal road surface 3D modeling, simulation and modeling steps are the same as in the previous embodiment, and will not be repeated here.

[0136] By implementing the above vehicle control method, not only can the beneficial effects described in the previous embodiment be obtained, but the computational burden can also be transferred to the cloud server, giving full play to the advantages of 5G technology and cloud computing power; at the same time, local computing power can be better allocated to the ECU (electronic controller unit) modules of the engine, windows and infotainment system, minimizing system redundancy and waste, and achieving better autonomous management of the whole vehicle.

[0137] The present invention also provides a server configured to execute the vehicle control method described above, wherein: the cloud server receives road and driving data sent by the vehicle via 5G technology, and executes the method as described above. Figure 6 The process shown transmits the calculation results to the vehicle via 5G technology, and can also share them with vehicles traveling nearby to ensure that the vehicle can pass safely.

[0138] like Figure 6 The implementation flowchart of the vehicle control method on the cloud server is shown, and the specific steps are as follows:

[0139] S601: Road Anomaly Detection. After receiving information about the road surface ahead, the cloud server calculates the deviation information of the road surface point cloud data to determine whether there are any anomalies on the road surface ahead of the vehicle. Specifically, it can use a lidar to emit a detection signal (laser beam), and compare the received signal reflected back from the target (target echo) with the emitted signal to calculate the deviation distance between the point cloud points and the road surface. If the number of points with abnormal deviation distances exceeds a preset threshold, it is considered an abnormal road surface; if it is below the preset threshold, it is considered a normal road surface. Then, the anomaly is identified.

[0140] S602: Road Surface Anomaly Detection. If an anomaly is detected, the image data captured by the binocular camera is further input into a pre-trained deep learning model to identify the type of road surface anomaly. This deep learning model has the ability to classify road conditions such as potholes, continuous speed bumps, gravel, and obstacles.

[0141] S603: 3D Road Surface Modeling. 3D modeling of abnormal road surfaces can be performed using binocular stereo vision fused with LiDAR data for reconstruction. The binocular stereo vision reconstruction method simulates the stereo imaging principle of human eyes, with left and right cameras simultaneously capturing images of an object in the scene. Through trigonometric relationships and the principle of parallax, the coordinate information of points on the object's surface from the same viewpoint of both cameras is obtained, thus constructing the object's position and shape. LiDAR 3D reconstruction involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal to obtain relevant information about the target, such as distance, azimuth, height, attitude, and even shape parameters. By fusing LiDAR measurement data, problems such as image matching errors and positioning errors in binocular stereo vision 3D reconstruction are corrected, resulting in a more accurate 3D reconstruction model of the abnormal road surface and generating a 3D road surface map model.

[0142] S604: Vehicle Kinematics and Dynamics Simulation. Based on the obtained 3D model of the abnormal road surface, obstacle distances, angles, and vehicle speed, Simulnk software is used to conduct vehicle kinematics and dynamics simulation experiments. This indirectly simulates relevant parameters of the vehicle actually passing through abnormal road surfaces, such as vehicle speed, steering angle, suspension vertical force, camber angle, toe angle, vertical force, suspension geometry, center of gravity sideslip angle, and path curvature. Based on these parameters and the multi-degree-of-freedom kinematic model and vehicle dynamics model, a vehicle suspension adjustment scheme is determined. The multi-degree-of-freedom kinematic model and vehicle dynamics model can be, for example, simulation models provided by Simulnk software. In specific implementations, technicians can flexibly set these parameters according to the vehicle model and relevant design parameters; this will not be elaborated upon here.

[0143] S605: Based on 5G communication technology, the vehicle's suspension adjustment commands are sent to the vehicle in real time for proactive suspension adjustments, ensuring smooth driving operation, vehicle passability, comfort, and safety. Simultaneously, based on real-time sharing of road anomaly data via 5G, nearby drivers are alerted to potential abnormalities ahead and are advised to drive cautiously.

[0144] The above steps are an optional implementation method for cloud servers. In fact, the implementation methods described above can be combined, and the specific combination methods will not be repeated here.

[0145] It is understood that while this application discloses the use of cloud processors for data processing and sharing, it is not limited to cloud processing methods. Data processing and sharing can also be performed using onboard computing power, provided the vehicle's computing power allows.

[0146] In addition, the present invention provides a vehicle configured to perform the vehicle control method described above; and the present invention also provides a system comprising the server described above and the vehicle described above.

[0147] The cloud server, vehicle, and system that implement the vehicle control method of the present invention described above have the same beneficial effects as the vehicle control method described above, and will not be repeated here.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0154] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0156] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings; however, the present invention is not limited thereto. Within the scope of the inventive concept, various simple modifications can be made to the technical solutions of the present invention, including combinations of various specific technical features in any suitable manner. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately. However, these simple modifications and combinations should also be considered as the content disclosed in the present invention and are all within the protection scope of the present invention.

Claims

1. A road surface detection method, characterized in that, The method includes: Acquire first sensor data of the road surface on which the vehicle is traveling; Based on the data from the first sensor, a first judgment result is obtained regarding whether there is an anomaly on the road surface. If the first judgment result indicates that there is an anomaly on the driving road surface, the second sensor data of the driving road surface shall be acquired; Based on the data from the second sensor, a second judgment result is obtained regarding whether there is an anomaly on the road surface; and Based on the first judgment result and the second judgment result, determine whether there is any abnormality on the driving road surface; The first sensor data and the second sensor data were collected using different types of sensors. When the first sensor data or the second sensor data is road surface point cloud data, the step of obtaining a first judgment result on whether there is an anomaly on the driving road surface based on the first sensor data, or the step of obtaining a second judgment result on whether there is an anomaly on the driving road surface based on the second sensor data, includes: Obtain the three-dimensional coordinates of all road points in the road point cloud data; Calculate the plane S of the driving road surface; Traverse all the road surface points. If the distance between the road surface point and the plane S exceeds a first preset threshold, then the road surface point is an anomaly. The cumulative number of abnormal points is calculated. If the number of abnormal points is greater than a second preset threshold, it is determined that there is an abnormality in the road surface; otherwise, there is no abnormality.

2. The road surface detection method according to claim 1, characterized in that, Determining whether there is an abnormality on the road surface includes: Logical judgment method: When both the first judgment result and the second judgment result indicate that the road surface is abnormal, it is determined that the road surface is abnormal; or, Alternatively, a logical judgment method may be used: if either the first judgment result or the second judgment result indicates that the road surface is abnormal, then the road surface is determined to be abnormal.

3. The road surface detection method according to claim 2, characterized in that: Based on one or more of the vehicle's computing power configuration, road surface and weather conditions, driving mode, and cloud server configuration, select to execute either the AND logic judgment method or the OR logic judgment method.

4. The road surface detection method according to claim 1, characterized in that: The first sensor data and the second sensor data are collected by the vehicle-mounted sensors and transmitted to the cloud server; and After the cloud server obtains the first sensor data and the second sensor data, it executes the following: Based on the data from the first sensor, a first judgment result is obtained regarding whether there is an anomaly on the road surface; simultaneously, based on the data from the second sensor, a second judgment result is obtained regarding whether there is an anomaly on the road surface. and The road surface is deemed abnormal if any of the following conditions are met: Both the first and second judgment results indicate that there is an anomaly on the road surface; or, The first or second judgment result indicates that there is an anomaly on the road surface.

5. The road surface detection method according to claim 1, characterized in that: The first sensor data is either road surface point cloud data or road surface image data; and The second sensor data is either road surface point cloud data or road surface image data.

6. The road surface detection method according to claim 5, characterized in that: The road surface point cloud data is acquired through one of the following methods: lidar, millimeter-wave radar, or ultrasonic radar; and The road surface image data is acquired using one of the following: a monocular camera, a binocular camera, a tricular camera, or a surround-view camera.

7. The road surface detection method according to claim 1, characterized in that, The method for calculating the plane S of the driving road surface includes: Obtain the normal vector of the chassis plane of the vehicle, and use the normal vector of the chassis plane of the vehicle as the normal vector (a, b, c) of the plane S. Calculate the mean point A0(x0, y0, z0) of all road surface points within the road surface point cloud data; and According to the point method formula for a plane: a(x-x0)+b(y-y0)+c(z-z0)=0, the equation of the plane S is obtained as: ax+by+cz+d=0, where x, y, z are the coordinates of a point on the plane S, and a, b, c, d are the parameters of the equation of the plane S.

8. The road surface detection method according to claim 5, characterized in that, When the first sensor data or the second sensor data is the road surface image data, obtaining a first judgment result regarding whether there is an anomaly on the road surface based on the first sensor data, or obtaining a second judgment result regarding whether there is an anomaly on the road surface based on the second sensor data, includes: The road surface image data is input into the trained deep learning model to obtain the judgment result of whether there is an anomaly on the driving road surface output by the deep learning model.

9. A vehicle control method, characterized in that: When the road surface detection method according to any one of claims 1-8 determines that the road surface on which the vehicle is traveling is an abnormal road surface, the following steps are performed: The abnormal road surface is modeled in three dimensions to obtain a three-dimensional model of the abnormal road surface. Based on the three-dimensional model of the abnormal road surface, obtain the driving parameters of the vehicle passing through the abnormal road surface; Based on the driving parameters, kinematic model, and vehicle dynamics model, a vehicle suspension adjustment scheme is obtained; and Control the vehicle to execute the vehicle suspension adjustment scheme.

10. The vehicle control method according to claim 9, characterized in that, The process of performing three-dimensional modeling on the abnormal road surface to obtain a three-dimensional model of the abnormal road surface includes: Calculate the image data feature points and their three-dimensional coordinates based on the image data; Based on the road surface point cloud data, calculate the feature points of the road surface point cloud data and their three-dimensional coordinates; Matching the feature points in the image data with the feature points in the road surface point cloud data, and excluding invalid points in both, yields the matching points and their three-dimensional coordinates; and By merging the matching points and their three-dimensional coordinates, the three-dimensional model of the abnormal road surface is obtained.

11. The vehicle control method according to claim 9, characterized in that, The simulation parameters for vehicle travel on the abnormal road surface, based on the three-dimensional model of the abnormal road surface, include: The abnormal road surface 3D model, vehicle driving data, and obstacle distance and angle information are acquired and input into the simulation tool to simulate the vehicle's driving parameters when passing through the abnormal road surface.

12. A server configured to perform the road surface detection method according to any one of claims 1-8 and / or the vehicle control method according to any one of claims 9-11.

13. A vehicle configured to perform the road surface detection method according to any one of claims 1-8 and / or the vehicle control method according to any one of claims 9-11.

14. A machine-readable storage medium storing instructions for causing a machine to perform the road surface detection method according to any one of claims 1-8 and / or the vehicle control method according to any one of claims 9-11.

Images