A safe driving avoidance method for commercial vehicles based on multi-source spatiotemporal data fusion

Abstract

The present invention provides a commercial vehicle safe driving avoidance method with multi-source spatiotemporal data fusion, which includes the following steps: based on the image recognition result, the system obtains the sampling data of the multi-source sensors and classifies and fuses them. An obstacle trajectory prediction model is constructed, and the predicted obstacle motion trajectory is obtained based on the fused data, and then the collision situation is judged based on the vehicle motion situation and the predicted obstacle motion trajectory; when there is a collision situation, the on-board auxiliary driving system or the automatic driving system controls the vehicle to perform obstacle avoidance action. The present invention obtains the image of the preceding vehicle in real time to provide a warning of falling objects, and combines the multi-source sensor data of the vehicle peripherals to predict the movement trajectory of the obstacle in front of the vehicle, judges whether it is necessary to avoid according to the collision situation, and formulates a vehicle obstacle avoidance plan, so that the key positions of the vehicle can avoid being hit by obstacles, thereby improving the safety of driving and vehicle peripherals.

Description

Technical Field

[0001] The present invention relates to the field of safe driving technology, and in particular to a commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion. Background Art

[0002] Safe driving has always been a key issue that people pay attention to. It is not only related to the safety of the driver himself, but also to the safety of passengers and other people on the road. Therefore, how to improve driving safety has always been one of the main research directions of technicians.

[0003] Since autonomous or assisted operation of machines can reduce the impact of human errors on vehicle driving, assisted driving and autonomous driving technologies are developing rapidly in the field of safe vehicle driving. Their first application scenarios are mostly commercial vehicles. The main reason is that the operating efficiency, safety and fuel economy of commercial vehicles in unmanned driving scenarios are more advantageous than those of pure manual driving, and preset rules can be used to make them pay more attention to ride comfort and the safety of vehicle peripherals.

[0004] However, since existing roads are not designed based on the requirements of assisted driving or autonomous driving scenarios, real roads are complex and changeable. In a complex and changeable driving environment, driverless commercial vehicles face emergencies such as traffic obstacles ahead, which may lead to decision-making errors and safety issues.

[0005] At present, in the existing technology, during the assisted driving or unmanned driving process, emergencies may occur and cause damage to the vehicle. For example, when the vehicle is driving, various transport vehicles appear in front of it, and the transported objects may slide due to bumpy road conditions. For example, gravel, when it slides and hits weak parts such as sensors installed on the outside of the vehicle, such peripherals may be damaged, and the vehicle may be unable to continue to obtain external data or may cause significant interference to the collected data, which may cause the vehicle to interrupt the assisted or unmanned driving process. This not only endangers road safety, but also affects overall traffic efficiency.

[0006] In summary, a method needs to be studied to provide an avoidance strategy for commercial vehicles with assisted driving or unmanned driving, so as to protect the safety of passengers and vehicles. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a commercial vehicle safe driving avoidance method by fusing multi-source spatiotemporal data.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] The main steps are as follows:

[0010] The vehicle-mounted camera collects the image of the vehicle ahead within the preset distance in real time, and identifies the transported objects of the vehicle ahead through image recognition method;

[0011] When the transported object of the preceding vehicle belongs to a preset category, the system acquires sampling data of vehicle sensors, wherein the vehicle sensors include multi-source sensors and at least include a laser radar and a millimeter-wave radar; performs data preprocessing and feature information extraction on the sampling data to obtain effective sensor data;

[0012] Classifying the valid sensor data to obtain obstacle data and vehicle data; each obstacle in the obstacle data is configured with a unique obstacle identifier;

[0013] fusing the obstacle data based on the obstacle identifier to obtain an obstacle three-dimensional motion data set;

[0014] Construct obstacle trajectory prediction model;

[0015] A time series set of obstacle three-dimensional motion data sets within a preset time period is selected as a model input, and the model input is input into the obstacle trajectory prediction model to obtain obstacle prediction motion trajectory data;

[0016] Constructing a vehicle trajectory prediction model based on the vehicle data and obtaining vehicle predicted motion trajectory data;

[0017] Based on the vehicle predicted motion trajectory data and the three-dimensional space coordinate set of the vehicle sensor, and the obstacle predicted motion trajectory data, determine the collision situation;

[0018] When a collision is determined, the on-board assisted driving system or the automatic driving system controls the vehicle to perform obstacle avoidance actions.

[0019] Optionally, obstacle image information is also collected by the vehicle-mounted camera, and obstacle information is obtained by an image recognition method;

[0020] The obstacle categories are divided according to preset rules and the obstacle information, and the obstacle categories include at least three categories, namely, first category obstacles, second category obstacles and third category obstacles;

[0021] The first type of obstacle is a non-dangerous obstacle, and the obstacle mark is not added to the first type of obstacle;

[0022] The second type of obstacle is a limited danger obstacle, and the obstacle mark is added to the second type of obstacle;

[0023] The third type of obstacle is a dangerous obstacle, and the obstacle mark and the danger mark are added to the third type of obstacle.

[0024] Optionally, performing classification comparison when judging the collision situation includes the following steps:

[0025] When the third type of obstacle does not exist, only determining whether the vehicle sensor collides with the second type of obstacle;

[0026] When there is a third type of obstacle, first determine whether the vehicle body collides with the third type of obstacle; if the vehicle body collides with the third type of obstacle, then determine whether the vehicle sensor collides with the second type of obstacle or the third type of obstacle; if the vehicle body does not collide with the third type of obstacle, then determine only whether the vehicle sensor collides with the second type of obstacle;

[0027] When only the third type of obstacle exists, it is determined whether the vehicle body collides with the third type of obstacle.

[0028] Optionally, the obstacle three-dimensional motion data set includes obstacle identification, spatial position coordinates, distance from the vehicle body, azimuth and speed;

[0029] Clustering or segmenting the laser radar data to obtain point cloud data of obstacles;

[0030] The speed of the obstacle is extracted from the data of the millimeter wave radar through a target detection algorithm and a target tracking algorithm.

[0031] Optionally, the method for extracting the spatial position coordinates includes:

[0032] Based on the point cloud data, a set of points corresponding to the obstacle is selected, and coordinates of the points in the set are accumulated to obtain a coordinate accumulation sum;

[0033] The accumulated sum of the coordinates is divided by the total number of points in the set to obtain the coordinates of the center of gravity of the obstacle, and the coordinates of the center of gravity of the obstacle are used as the spatial position coordinates.

[0034] Optionally, the collision situation is determined by a distance calculation method;

[0035] The collision condition is that there is a collision time point, and the minimum distance between the obstacle trajectory point corresponding to the collision time point and the vehicle body or the vehicle-mounted sensor in the vehicle trajectory point is less than the safety threshold;

[0036] The safety threshold is greater than the maximum radius of the obstacle.

[0037] Optionally, the vehicle data further includes a vehicle driving plan, and a predicted driving state is obtained according to the vehicle driving plan;

[0038] The step of constructing the obstacle trajectory prediction model also includes the following steps:

[0039] When the predicted driving state is a uniform and straight state, the obstacle three-dimensional motion data set is a motion data set relative to the vehicle;

[0040] When the predicted driving state is a non-uniform straight state, the obstacle three-dimensional motion data set is converted into a motion data set relative to the ground by using a reference system conversion method.

[0041] Optionally, the obstacle trajectory prediction model is a long short-term memory network;

[0042] The obstacle trajectory prediction model includes at least an input layer, a first hidden layer and a fully connected layer.

[0043] Optionally, the obstacle avoidance action obtaining method comprises the following steps:

[0044] An objective function is constructed with the goal of minimizing the cost of vehicle action, and the vehicle body geometry parameters and the three-dimensional space coordinate set of the vehicle sensor are introduced into the objective function. The objective function is then solved by the gradient descent method to obtain the speed and acceleration of the vehicle in the obstacle avoidance action.

[0045] Optionally, the obstacle avoidance action includes vehicle body swinging and / or vehicle acceleration and deceleration.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The method of the present invention obtains the image of the vehicle in front in real time to give a warning of fallen objects (obstacles), and combines the multi-source sensor data of the vehicle peripherals to predict the movement trajectory of the obstacle appearing in front of the vehicle, and judges whether to avoid it according to the obstacle movement prediction and the vehicle movement prediction, and formulates a vehicle obstacle avoidance plan, so that the key position of the vehicle can be prevented from being hit by the obstacle, the vehicle assisted driving or unmanned driving process is not significantly affected, and the safety of driving and vehicle peripherals is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0052] In the description of the present invention, the meaning of "speed" is a conventional definition, and should be understood as a description including direction, unless otherwise clearly and specifically defined.

[0053] It is worth noting that the methods used in the present invention are all conventional methods unless otherwise specified; the raw materials and devices used are all conventional commercially available products, and their sources are not specifically limited unless otherwise specified.

[0054] It should also be noted that the following embodiments are described in a certain order in order to facilitate the understanding of the technical solution. Some steps can be performed simultaneously or in an adjusted order, and the description order should not be understood as a limiting condition.

[0055] like Figure 1 As shown, this embodiment provides a commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion, which mainly includes the following steps:

[0056] First of all, it should be explained that the vehicle used in this embodiment has an assisted driving system or an unmanned driving system, which is configured with corresponding common sensors, including but not limited to lidar, millimeter-wave radar, and vehicle-mounted cameras, and the manufacturer has pre-stored vehicle information in the system, including vehicle body geometry parameters (such as vehicle height, vehicle length, and vehicle contour data if necessary) and the size of each sensor and the installation position information relative to the vehicle.

[0057] During normal driving, the system calls the on-board camera to collect the image of the front vehicle within the preset vehicle distance in real time, that is, pay attention to the loading of goods in the front vehicle, and use the built-in image recognition algorithm of the system to quickly identify the transported objects of the front vehicle; the main purpose of this step is to warn of emergencies as described in the background technology, especially in the case of the front vehicle dropping transported objects within a certain vehicle distance range; because the splashes from the road surface during normal driving are generally small in volume or weight, and the larger weight or volume is generally the transported objects dropped by the front vehicle, such as: gravel, equipment, packaging boxes, etc., and this type of items can often cause significant damage or driving interference to the rear vehicle. Based on this, using the image recognition results within the non-safe range as the subsequent start condition can save system resources under safe driving conditions and ensure that potential dangers are paid attention to in advance. It should be noted that if the current vehicle has no exposed transported objects, such as cars or vans, the image cannot detect the transported objects, and there is no risk of the transported objects falling, so there is no need to start the subsequent process.

[0058] Under the above system operation, when the identified transported objects of the preceding vehicle belong to a preset category, such as gravel, glass, steel coils, packaging boxes and other dangerous goods, the system starts subsequent work to further obtain sampling data from other vehicle sensors. Among them, the vehicle sensors mentioned in this embodiment include multi-source sensors, which can be understood as multiple different types of sensors (i.e., data sources are diverse), and at least include the laser radar and millimeter-wave radar described above. The sampled data is preprocessed and feature information is extracted to obtain valid sensor data. When the system collects sensor data that an object has slipped, that is, the speed is inconsistent with that of the preceding vehicle, it can be judged that the object has slipped, thereby forming a dynamic obstacle, and the subsequent process can then continue.

[0059] The laser radar data includes a 3D point cloud of the scanned area, which contains the 3D spatial coordinates of each point; for example: (x1, y1, z1), (x2, y2, z2)...(x n ,y n ,z n). In the point cloud map, the position and shape of the gravel that has fallen from the front vehicle can be clearly seen, so as to obtain relevant data on sudden gravel that may cause damage to the vehicle. The laser radar acquisition cycle is 125ms-250ms. The data of the millimeter-wave radar includes the distance of each obstacle from the vehicle in the scanning area and the direction angle between it and the vehicle body; the millimeter-wave radar acquisition cycle is 125ms~250ms; for example: the front transport vehicle is 10 meters away and the speed is 20 kilometers per hour. The distance of one of the fallen gravel from the vehicle and the direction angle between the gravel and the vehicle body are 0.5 meters and 30 degrees respectively; the moving speed of the gravel can be calculated based on the last acquired data and the current acquired data. The above data is cleaned and simply calculated to obtain the aforementioned valid sensor data to ensure data quality and consistency.

[0060] The effective sensor data is classified. Since the overall data also includes the vehicle's own driving speed, acceleration, etc., the data is classified to obtain obstacle data and vehicle data; each obstacle in the obstacle data is configured with a unique obstacle identifier.

[0061] Optionally, obstacle image information is collected through the vehicle camera, and obstacle information is obtained through image recognition methods, such as YOLO, Faster R-CNN or SSD, so as to identify the target in the image and determine the position of the target in each image frame. Obstacle categories are divided according to preset rules and obstacle information, and the obstacle categories include at least three categories, namely, first-category obstacles, second-category obstacles and third-category obstacles.

[0062] The first type of obstacles are non-dangerous obstacles, that is, objects that will not cause harm to the vehicle body or sensors after colliding with the vehicle at the current speed, such as small-volume sand and gravel, plastic bags, paper products, beverage bottles, etc., and no obstacle mark is added to the first type of obstacles;

[0063] The second type of obstacles are limited danger obstacles, that is, objects that will not cause harm to the vehicle body after colliding with the vehicle at the current speed, but will affect the functions of at least some sensors, such as medium-sized sand and gravel, packaging boxes, screws, etc., and obstacle labels will be added to the second type of obstacles;

[0064] The third type of obstacles are dangerous obstacles, i.e. objects that will cause harm to the vehicle body and sensors after colliding with the vehicle at the current speed, such as large volumes of sand and gravel, building materials, engineering components, etc. Obstacle signs and danger signs are added to the third type of obstacles.

[0065] It should be noted that the “hazard” mentioned above is further defined as causing plastic deformation or fragmentation in the collision area, or causing irreversible damage to the structural parts in the area.

[0066] Therefore, the obstacle data is fused based on the obstacle identification to obtain the obstacle three-dimensional motion data set; specifically, taking the gravel that slipped off the previous vehicle as an example, each piece of gravel is configured with a unique obstacle identification, that is, the gravel ID, so that the data under the ID is integrated to form the obstacle three-dimensional motion data set. The obstacle three-dimensional motion data set includes obstacle identification, spatial position coordinates, distance from the vehicle body, azimuth and speed. The specific fusion method is to start with the gravel ID and splice each data to form a one-dimensional characteristic phasor, such as: gravel 1 [ID, three-dimensional spatial position coordinates, distance from the vehicle body, azimuth, speed].

[0067] The method for obtaining some data is as follows:

[0068] Three-dimensional spatial position coordinates: cluster or segment the lidar data and obtain the point cloud data of the obstacle; based on the point cloud data, select a set of points corresponding to the obstacle, accumulate the coordinates of the points in the set, and obtain the cumulative sum of the coordinates; divide the cumulative sum of the coordinates by the total number of points in the set to obtain the coordinates of the center of gravity of the obstacle, and use the coordinates of the center of gravity of the obstacle as the spatial position coordinates.

[0069] Speed: The speed of obstacles is extracted from the millimeter-wave radar data through target detection and target tracking algorithms. The displacement vector of each piece of sand and stone between adjacent frames can also be calculated through motion tracking algorithms (such as optical flow estimation algorithms), and the speed of the sand and stone can be determined based on the displacement vector.

[0070] It can be understood that the movement speed and direction data of each sand and stone include the movement speed (including movement direction) calculated from the lidar data, millimeter wave radar data, and camera data respectively, which can be compared, verified or averaged with each other to improve the accuracy of the data and prevent data errors caused by external factors when a single sensor is used, such as electromagnetic wave interference, light interference, etc.

[0071] Optionally, since the data directly collected by the sensor is generally relative quantity, that is, the vehicle body is used as the reference system, the general practice in the prior art is to convert the obstacle three-dimensional motion data set into a data set with the ground as the reference system through the coordinate system conversion method, and use this for subsequent comparison or judgment. Although the method of doing so is technically mature, and because vehicle navigation, automatic driving, etc. are all based on the ground reference system, it is beneficial to predict collision situations under a unified coordinate system (ground), however, such a step operation actually increases the amount of calculation of the system, and the vehicle is mostly in a stable driving state in scenes such as high speeds, so it is necessary to classify and process the collision judgment of obstacles.

[0072] Therefore, the aforementioned vehicle data also includes the vehicle driving plan acquired by the system, such as navigation information and automatic driving or assisted driving strategies, and obtains the predicted driving status in the future period based on the vehicle driving plan.

[0073] Based on this, an obstacle trajectory prediction model is constructed, which includes the following steps:

[0074] When the predicted driving state is a uniform and straight state, the obstacle three-dimensional motion data set is maintained as the motion data set relative to the vehicle;

[0075] When the predicted driving state is a non-uniform straight state, that is, there is acceleration, deceleration or curved motion, the three-dimensional motion data set of the obstacle is converted into a motion data set relative to the ground through a reference system conversion method according to conventional methods.

[0076] Afterwards, the obstacle three-dimensional motion data set within a preset time period is selected to form a set according to the time series within the time period, and this is used as the model input; the model input is input into the obstacle trajectory prediction model to obtain the obstacle prediction motion trajectory data.

[0077] For example, define the length of the preset time period as T, that is, use the data of the past T time steps as input features, and combine all the features of T time steps into an input sequence X;

[0078] Then X={x t-T+1 ,x t-T+2 ......x t}, where x t represents the feature vector of the tth time step, that is, the feature vector of the current time point, x t-T+1 It represents the feature vector of the t-T+1th time step, that is, the feature vector of the set time point; the time interval between each time step is the period of laser radar and millimeter wave radar data collection.

[0079] Among them, x t It includes the ID of each piece of gravel at the current time point, the three-dimensional spatial position coordinates, the distance between the gravel and the vehicle body, the direction angle between the gravel and the vehicle body, the moving speed and moving direction of the gravel, and is arranged in the order of the gravel ID.

[0080] Optionally, the obstacle trajectory prediction model constructed in this embodiment is a long short-term memory network, which includes at least an input layer, a first hidden layer and a fully connected layer.

[0081] Among them, the input layer is used to input the input sequence into the first hidden layer;

[0082] The first hidden layer is used to capture dynamic features in the input sequence;

[0083] The fully connected layer is used to map the output of the first hidden layer to the target variable space and predict the spatial position of each gravel at the future moment; the future moment can be defined by the time point of laser radar and millimeter wave radar data collection, and the gravel trajectory prediction model can be updated through the subsequent real-time collected data;

[0084] The calculation formula for the first hidden layer is:

[0085] f t =σ(W1·[h t-1 ,x t ]+b1)

[0086] i t =σ(W2·[h t-1 ,x t ]+b2)

[0087]

[0088] O t =σ(W4·[h t-1 ,x t ]+b4)

[0089] h t =O t *tanh(c t )

[0090] In the formula, f t represents the forget gate vector at the tth time step, x t represents the feature vector at the tth time step, i t represents the input gate vector at the tth time step, h t-1 represents the hidden state vector at the t-1th time step, represents the candidate cell state vector at the tth time step, c t represents the cell state vector at the tth time step, c t-1 represents the cell state vector at the t-1th time step, O t represents the output gate vector at the tth time step, h t represents the hidden state vector output at the tth time step, W1, W2, W3 and W4 all represent trainable weight parameters, b1, b2, b3 and b4 all represent trainable bias parameters, tanh represents the activation function of the hyperbolic tangent function, and σ represents the activation function of the sigmoid function.

[0091] The calculation formula of the fully connected layer is:

[0092] Y t+s =W5h t +b5

[0093] Where Y t+s It represents the gravel position vector at the t+sth time step, contains the three-dimensional spatial position coordinates of each gravel, and has 1≤s≤K; W5 and b5 represent the trainable weight parameter and bias parameter respectively.

[0094] The model can be trained through the event data of actual sliding and colliding with the vehicle behind in the past. The real data can be obtained from the data storage device on the accident vehicle. At the same time, in order to increase the total number of samples, simulation analysis results can be introduced for training, that is, the sliding and collision of obstacles of different sizes, weights, speeds, and positions can be simulated in the kinematic simulation software, and this can be used as the corresponding model training data set.

[0095] It is worth noting that since the obstacle three-dimensional motion data set was classified in the early stage, two sets of models need to be prepared accordingly. Therefore, the training set is also divided into two categories, one is the training set with the rear car as the reference system, and the other is the training set with the ground as the reference system.

[0096] The predicted motion trajectory data of each sandstone in the model output includes the three-dimensional spatial position coordinate data of each sandstone at K time points after the current time point, expressed as: Z = {z t+1 ,z t+2 ......z t+K}; where z t+1 The feature vector z represents the concatenation of the three-dimensional spatial position coordinates of each sandstone at the first time point after the current time point. t+K It represents the feature vector formed by splicing the three-dimensional spatial position coordinates of each sandstone at the Kth time point after the current time point; the time interval between each time point is the period of laser radar and millimeter wave radar data collection.

[0097] The vehicle trajectory prediction model is constructed based on the vehicle data, and the vehicle prediction motion trajectory data is obtained. Since the vehicle conditions are discussed in a classified manner, when the predicted driving state of the vehicle within the predetermined time period is a uniform and straight state, the vehicle is used as the reference system, and the model output result is (0,0,0); when the driving state of the vehicle within the predetermined time period is a non-uniform and straight state, the vehicle prediction motion trajectory data is output based on the vehicle driving plan.

[0098] The collision situation is judged based on the vehicle predicted motion trajectory data and the three-dimensional spatial coordinate set of the vehicle sensor and the obstacle predicted motion trajectory data.

[0099] Optionally, classification comparison is performed when determining the collision situation, including the following steps:

[0100] When there is no third-type obstacle, only determine whether the vehicle sensor collides with the second-type obstacle;

[0101] When there is a third type of obstacle, first determine whether the vehicle body collides with the third type of obstacle; if the vehicle body collides with the third type of obstacle, then determine whether the vehicle sensor collides with the second type of obstacle or the third type of obstacle; if the vehicle body does not collide with the third type of obstacle, then only determine whether the vehicle sensor collides with the second type of obstacle;

[0102] When only the third type of obstacle exists, it is determined whether the vehicle body collides with the third type of obstacle.

[0103] Among them, since the first type of obstacles are not initially assigned obstacle identification, they do not enter the subsequent process, so there is no need to perform collision judgment.

[0104] Furthermore, the collision situation is determined by a distance calculation method;

[0105] The collision condition is that there is a collision time point, and the minimum distance between the obstacle trajectory point corresponding to the collision time point and the vehicle body or vehicle-mounted sensor in the vehicle trajectory point is less than the safety threshold;

[0106] The safety threshold is greater than the maximum radius of the obstacle, such as the maximum radius of sand and gravel + h max ,h max It is the maximum height of all sensors protruding from the vehicle body contour.

[0107] When a collision is determined, the on-board assisted driving system or the automatic driving system controls the vehicle to perform obstacle avoidance actions.

[0108] Optionally, the obstacle avoidance action obtaining method includes the following steps:

[0109] An objective function is constructed with the goal of minimizing the cost of vehicle action, and the vehicle geometry parameters and the three-dimensional spatial coordinate set of the vehicle sensor are introduced into the objective function. The objective function is then solved by the gradient descent method to obtain the speed and acceleration of the vehicle during the obstacle avoidance action.

[0110] Specifically, the cost of the vehicle action is defined as the sum of the square of the velocity change and the square of the acceleration to ensure that the drastic changes of the vehicle are minimized when implementing the evasive action. Introducing the vehicle body geometry parameters in the objective function, such as the length, width and height of the vehicle body, can help determine the bounding box of the vehicle, so as to more accurately evaluate the spatial relationship between the vehicle and the obstacle in collision detection. At the same time, the three-dimensional spatial coordinate set of the vehicle sensor is brought in to consider the impact of the obstacle on the vehicle action in the objective function. For example, if the obstacle is calculated to be close to the sensor position of the vehicle, the objective function can adjust the vehicle action accordingly to avoid the area first. Further, in order to solve the objective function to obtain the speed and acceleration of the vehicle in the obstacle avoidance action, the gradient descent method is used; first, the initial value is determined, such as setting the initial speed and acceleration according to the current driving state of the vehicle. During the execution of the gradient descent method, an adaptive step size is selected, which can be determined by experiment or experience and dynamically adjusted according to the operation of the algorithm; constraints are set, such as the maximum speed and maximum acceleration limit of the vehicle. In addition, the constructed objective function and solution algorithm must be fully verified and tested to ensure that they can stably and accurately calculate the speed and acceleration in obstacle avoidance actions under various scenarios and input data, thereby improving the robustness of the algorithm; the convergence conditions of the algorithm must be set, such as stopping the algorithm when the value change of the objective function is less than a specified threshold or a certain number of iterations is reached.

[0111] Therefore, the speed and acceleration required for the vehicle avoidance action can be solved according to the above method. Since some of the actions will include lateral speed, the actual reaction results of the obstacle avoidance action include vehicle body swing and / or vehicle acceleration and deceleration, wherein the vehicle body swing can be achieved by turning the steering wheel to the left or right to set an angle, and turning the steering wheel can change the vehicle's moving trajectory, thereby changing the trajectory of the vehicle's expected collision position, thereby avoiding the collision or at least not colliding with the sensor. Specific parameters such as the angle and time of turning the steering wheel can be calculated based on the actual vehicle speed and vehicle model.

[0112] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion, characterized by: The steps include: The vehicle-mounted camera collects the image of the vehicle ahead within the preset distance in real time, and identifies the transported objects of the vehicle ahead through image recognition method; When the transported object of the preceding vehicle belongs to a preset category, the system acquires sampling data of vehicle sensors, wherein the vehicle sensors include multi-source sensors and at least include laser radar and millimeter wave radar; performs data preprocessing and feature information extraction on the sampling data to obtain effective sensor data; The valid sensor data is classified to obtain obstacle data and vehicle data; each obstacle in the obstacle data is configured with a unique obstacle identifier; the vehicle-mounted camera collects obstacle image information and obtains obstacle information through an image recognition method; The obstacle categories are divided according to preset rules and the obstacle information, and the obstacle categories include at least three categories, namely, first category obstacles, second category obstacles and third category obstacles; The first type of obstacle is a non-dangerous obstacle, and the obstacle mark is not added to the first type of obstacle; The second type of obstacle is a limited danger obstacle, and the obstacle mark is added to the second type of obstacle; The third type of obstacle is a dangerous obstacle, and the obstacle mark and the danger mark are added to the third type of obstacle; fusing the obstacle data based on the obstacle identifier to obtain an obstacle three-dimensional motion data set; Construct obstacle trajectory prediction model; A time series set of obstacle three-dimensional motion data sets within a preset time period is selected as a model input, and the model input is input into the obstacle trajectory prediction model to obtain obstacle prediction motion trajectory data; Constructing a vehicle trajectory prediction model based on the vehicle data and obtaining vehicle predicted motion trajectory data; Based on the vehicle predicted motion trajectory data and the three-dimensional space coordinate set of the vehicle sensor, and the obstacle predicted motion trajectory data, a collision situation is judged; and classification comparison is performed when judging the collision situation, including the following steps: When the third type of obstacle does not exist, only determining whether the vehicle sensor collides with the second type of obstacle; When there is a third type of obstacle, first determine whether the vehicle body collides with the third type of obstacle; if the vehicle body collides with the third type of obstacle, then determine whether the vehicle sensor collides with the second type of obstacle or the third type of obstacle; if the vehicle body does not collide with the third type of obstacle, then determine only whether the vehicle sensor collides with the second type of obstacle; When only the third type of obstacle exists, determining whether the vehicle body collides with the third type of obstacle; When it is determined that there is a collision, the vehicle-mounted auxiliary driving system or the automatic driving system controls the vehicle to perform obstacle avoidance actions; the obstacle avoidance action obtaining method comprises the following steps: An objective function is constructed with the goal of minimizing the cost of vehicle action, and the vehicle body geometry parameters and the three-dimensional space coordinate set of the vehicle sensor are introduced into the objective function. The objective function is then solved by the gradient descent method to obtain the speed and acceleration of the vehicle in the obstacle avoidance action.

2. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 1 is characterized by: The obstacle three-dimensional motion data set includes obstacle identification, spatial position coordinates, distance from the vehicle body, azimuth and speed; Clustering or segmenting the laser radar data to obtain point cloud data of obstacles; The speed of the obstacle is extracted from the data of the millimeter wave radar through a target detection algorithm and a target tracking algorithm.

3. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 2 is characterized by: The method for extracting the spatial position coordinates includes: Based on the point cloud data, a set of points corresponding to the obstacle is selected, and coordinates of the points in the set are accumulated to obtain a coordinate accumulation sum; The accumulated sum of the coordinates is divided by the total number of points in the set to obtain the coordinates of the center of gravity of the obstacle, and the coordinates of the center of gravity of the obstacle are used as the spatial position coordinates.

4. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 3 is characterized by: The method for determining the collision situation is a distance calculation method; The collision condition is that there is a collision time point, and the minimum distance between the obstacle trajectory point corresponding to the collision time point and the vehicle body or the vehicle-mounted sensor in the vehicle trajectory point is less than the safety threshold; The safety threshold is greater than the maximum radius of the obstacle.

5. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 4 is characterized by: The vehicle data also includes a vehicle driving plan, and a predicted driving state is obtained according to the vehicle driving plan; The step of constructing the obstacle trajectory prediction model also includes the following steps: When the predicted driving state is a uniform and straight state, the obstacle three-dimensional motion data set is a motion data set relative to the vehicle; When the predicted driving state is a non-uniform straight state, the obstacle three-dimensional motion data set is converted into a motion data set relative to the ground by using a reference system conversion method.

6. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 1 is characterized by: The obstacle trajectory prediction model is a long short-term memory network; The obstacle trajectory prediction model includes at least an input layer, a first hidden layer and a fully connected layer.

7. The commercial vehicle safe driving avoidance method based on multi-source spatiotemporal data fusion according to claim 1 is characterized by: The obstacle avoidance action includes vehicle body swinging and / or vehicle acceleration and deceleration.

Images