An automatic driving obstacle intention prediction and avoidance method and system

Abstract

The present application relates to the technical field of automatic driving, in particular to an automatic driving obstacle intention prediction and avoidance method and system. The method comprises the steps of acquiring image data and three-dimensional point cloud data in real time, extracting static features and dynamic features respectively, using an attention mechanism to weight and fuse to form a fusion feature vector, inputting the fusion feature vector into a plurality of independently trained deep learning models, using model difference to weight and fuse the output results to obtain a predicted intention. Based on the predicted intention, the association constraints of the running state and the input control are set, an avoidance strategy model is established with path length and time cost as double optimization objectives, the input control is continuously updated within the prediction time domain until the optimization objective value is iterated to the minimum, the time sequence of the avoidance strategy is output, and the final execution instruction is generated in combination with the real-time running state. The present application can break through the bottleneck of the traditional system based on static features and preset rules, improve the driving safety redundancy, and significantly optimize the road traffic efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving obstacle intention prediction and avoidance method and system. BACKGROUND

[0002] In the field of automatic driving, accurate identification and efficient avoidance of surrounding obstacles are crucial for ensuring driving safety and improving road traffic efficiency. Existing obstacle identification methods mainly include, but are not limited to, relying on data collected by sensors such as radar, lidar, and cameras, as well as image recognition techniques based on these data. These techniques, while to some extent enabling detection and localization of obstacles, are often limited to analysis of single sensor data or focus only on static identification of obstacle appearance features.

[0003] For example, radar sensors can accurately measure the distance and relative speed of obstacles, but they have limitations in identifying specific types, shapes, and textures of obstacles. Camera sensors, on the other hand, can capture rich visual information, including color, shape, and texture of obstacles, but their performance can be significantly affected in complex environments such as night, rain, and fog. In addition, existing image recognition techniques can identify the appearance features of obstacles, but often lack a deep understanding of the dynamic behavior of obstacles, making it difficult to accurately predict their future motion trajectories and intentions. SUMMARY

[0004] The technical problem to be solved by the embodiments of the present application is to provide an automatic driving obstacle intention prediction and avoidance method and system to solve the problem that existing technologies cannot accurately identify obstacles in an automatic driving environment and the avoidance strategy is unreasonable.

[0005] The present application discloses an automatic driving obstacle intention prediction and avoidance method, comprising:

[0006] real-time acquisition of image data and three-dimensional point cloud data of obstacles in front during automatic driving;

[0007] extracting static features of the obstacles from the image data and dynamic features of the obstacles from the three-dimensional point cloud data;

[0008] using an attention mechanism to weight the static features and the dynamic features respectively, and fusing the weighted static features and the dynamic features to obtain a fusion feature vector;

[0009] inputting the fusion feature vector into a plurality of independently trained deep learning models, weighting and fusing the output results of the plurality of deep learning models based on the model differences of the plurality of deep learning models to obtain the predicted intention of the obstacles;

[0010] setting an associated constraint of a vehicle operating state and an input control according to the predicted intention, and establishing an avoidance strategy model with path length and time cost as double optimization objectives;

[0011] continuously updating the input control in a prediction time domain until the double optimization objective value of the avoidance strategy model is iterated to a minimum within the associated constraint, and outputting a time sequence of the input control to constitute an avoidance strategy;

[0012] acquiring the real-time operating state of the vehicle in the automatic driving process, combining the avoidance strategy and the real-time operating state to generate a final vehicle execution instruction.

[0013] Optionally, the image data and three-dimensional point cloud data of the front obstacle in the automatic driving process are acquired, including:

[0014] real-time acquisition of image data and three-dimensional point cloud data of the front obstacle in the automatic driving process, the image data containing appearance information of the obstacle, and the three-dimensional point cloud data containing shape and position information of the obstacle;

[0015] preprocessing the acquired image data, including removing noise from the image data using wavelet transform, and enhancing the contrast of the image data using histogram equalization, the function expression of the image data preprocessing being:

[0016]

[0017] In the formula, denotes the preprocessed image data, denotes the image data before preprocessing, denotes a composite transformation function of noise removal and contrast enhancement, denotes residual noise.

[0018] Optionally, the static feature of the obstacle is extracted from the image data, including:

[0019] extracting the static feature of the obstacle from the preprocessed image data using an appearance flow network constructed by multiple convolution layers, the static feature including shape, texture, and color histogram of the obstacle, and the function expression of the appearance flow network extracting the static feature being:

[0020]

[0021] In the formula, denotes a feature map output by the appearance flow network, denotes a nonlinear activation function, denotes a weight matrix of a convolution kernel, image data representing input appearance flow network, denotes a convolution bias term.

[0022] Optionally, the method of extracting the dynamic features of the obstacle from the three-dimensional point cloud data comprises:

[0023] extracting the dynamic features of the obstacle from the three-dimensional point cloud data using a motion flow network constructed by a gating cycle, the dynamic features comprising the speed, direction and acceleration of the obstacle, the function expression of the motion flow network for extracting the dynamic features being:

[0024]

[0025]

[0026]

[0027] wherein, denotes a reset gate, denotes a reset gate weight, denotes a reset gate bias term, denotes an update gate, denotes an update gate weight, denotes an update gate bias term, denotes a candidate hidden state, denotes a candidate hidden state weight, denotes a candidate hidden state bias term, denotes a nonlinear activation function, denotes a previous time step hidden state, denotes three-dimensional point cloud data of a current time step input, denotes a hyperbolic tangent activation function, denotes element-wise multiplication.

[0028] Optionally, the method of automatically predicting and avoiding obstacles for autonomous driving further comprises a method of fusing the static features and the dynamic features, comprising:

[0029] defining a first weight matrix and a first bias term of an attention mechanism, mapping the static features into a calculation space of attention weights using the defined first weight matrix, and obtaining the attention weights of the static features through a normalization function calculation, the calculation function expression of the static feature attention weights being:

[0030]

[0031] a second weight matrix and a second bias term defining an attention mechanism, mapping the dynamic feature into a calculation space of attention weights using the defined second weight matrix, and calculating the attention weight of the dynamic feature through a normalization function, the calculation function expression of the dynamic feature attention weight being:

[0032]

[0033] wherein, denotes the attention weight of the static feature, denotes the first weight matrix, denotes the static feature, denotes the first bias term, denotes the attention weight of the dynamic feature, denotes the second weight matrix, denotes the dynamic feature, denotes the second bias term, denotes the normalization function;

[0034] concatenating the weighted static feature and the dynamic feature to obtain the fusion feature vector, the function expression of the concatenation of the static feature and the dynamic feature being:

[0035]

[0036] wherein, denotes the fusion feature vector after concatenation, denotes element multiplication.

[0037] Optionally, the automatic driving obstacle intention prediction and avoidance method further comprises a method of predicting the fusion feature vector using a plurality of deep learning models to obtain the predicted intention, comprising:

[0038] obtaining the fusion feature vectors of a plurality of known intention obstacles in a historical automatic driving process and using them as training data;

[0039] using a plurality of deep learning models each constructed by a convolutional neural network, a fully connected layer and a nonlinear activation function, independently training a plurality of deep learning models using different initialization parameters and training data;

[0040] inputting the fusion feature vectors of the obstacles obtained in real time into a plurality of trained deep learning models respectively, and calculating the weight of the output result of each deep learning model using a random forest algorithm;

[0041] The output result of each deep learning model is multiplied by a corresponding weight, and the weighted output results of all the deep learning models are added and fused to obtain a final predicted intention, and a function expression of the weighted fusion of the plurality of deep learning models is:

[0042]

[0043] In the formula, represents the final predicted intention, represents the output result of the i-th deep learning model, represents the weight of the i-th deep learning model.

[0044] Optionally, the associated constraints of the vehicle operating state and the input control are set according to the predicted intention, and an avoidance strategy model is established with path length and time cost as dual optimization objectives, including:

[0045] The association of the operating state and the input control in the automatic driving process of the vehicle is set according to the dynamic characteristics of the vehicle and the road constraints, and a function expression of the association of the operating state and the input control is:

[0046]

[0047] In the formula, represents the vehicle operating state at time t, represents the vehicle input control at time t, represents a system dynamics model of the vehicle;

[0048] The constraints of the operating state and the input control in the automatic driving process of the vehicle are set according to the predicted intention of the obstacle, and a function expression of the constraints of the operating state and the input control is:

[0049]

[0050] In the formula, represents a set of constraint conditions, represents the predicted intention of the obstacle at time t;

[0051] An avoidance strategy model in a prediction time domain is established with path length and time cost as dual optimization objectives, and a function expression of the avoidance strategy model is:

[0052]

[0053] In the formula, represents a dual optimization objective value output by the avoidance strategy model, and represents a weight coefficient, and T represents a prediction time domain.

[0054] Optionally, the running state of the vehicle in the automatic driving process is acquired in real time, and a final vehicle execution instruction is generated in combination with the avoidance strategy and the real-time running state, comprising:

[0055] According to the avoidance strategy, a desired running state of the vehicle is determined, and a current actual running state of the vehicle in the automatic driving process is collected and acquired;

[0056] According to the desired running state and the actual running state, a target acceleration for vehicle control execution is calculated, and the calculation function expression of the target acceleration is:

[0057]

[0058] In the formula, The target acceleration for vehicle control execution is represented by a, The desired speed is represented by v, The desired acceleration is represented by a, The current actual speed of the vehicle is represented by v, The current actual acceleration of the vehicle is represented by a, The proportional gain coefficient is represented by k p, The differential gain coefficient is represented by k d.

[0059] According to the calculated target acceleration, a corresponding vehicle control instruction is output.

[0060] The application also discloses a prediction and avoidance system, which adopts the automatic driving obstacle intention prediction and avoidance method described above, and the system comprises:

[0061] A data acquisition module is configured to acquire image data and three-dimensional point cloud data of a front obstacle in real time during automatic driving;

[0062] A feature extraction module is configured to extract static features of the obstacle from the image data and dynamic features of the obstacle from the three-dimensional point cloud data;

[0063] A feature fusion module is configured to weight the static features and the dynamic features respectively by using an attention mechanism, and fuse the weighted static features and the dynamic features to obtain a fusion feature vector;

[0064] An intention prediction module is configured to input the fusion feature vector into a plurality of independently trained deep learning models, weight and fuse output results of the plurality of deep learning models based on model differences of the plurality of deep learning models to obtain a predicted intention of the obstacle;

[0065] A model establishing module is configured to set an associated constraint of a vehicle operating state and an input control according to the predicted intention, and to establish an avoidance strategy model with path length and time cost as double optimization targets;

[0066] An avoidance strategy generating module is configured to continuously update the input control in a prediction time domain until the double optimization target value of the avoidance strategy model is iterated to a minimum within the associated constraint, and to output a time sequence of the input control to constitute an avoidance strategy.

[0067] A control instruction generating module is configured to acquire the real-time operating state of the vehicle in an automatic driving process, and to generate a final vehicle execution instruction in combination with the avoidance strategy and the real-time operating state.

[0068] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the automatic driving obstacle intention prediction and avoidance method.

[0069] Compared with the prior art, the automatic driving obstacle intention prediction and avoidance method and system provided by the application has the following advantages:

[0070] By acquiring real-time image data and three-dimensional point cloud data, static features and dynamic features are extracted respectively, and the fusion feature vector is formed by weighting and fusing the features after using the attention mechanism. The fusion feature vector is input into multiple independently trained deep learning models, and the predicted intention is obtained by using the model difference to weight and fuse the output results. The associated constraint of the operating state and the input control is set based on the predicted intention, the avoidance strategy model is established with path length and time cost as double optimization targets, the input control is continuously updated in the prediction time domain until the optimization target value is iterated to a minimum, the time sequence of the avoidance strategy is output, and the final execution instruction is generated in combination with the real-time operating state. The application solves the problems of inaccurate obstacle identification, insufficient intention prediction and unreasonable avoidance strategy in the prior art, realizes more accurate intention prediction, more flexible constraint and optimization target setting, more efficient iterative control, and significantly improves the driving safety and road traffic efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0071] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. In the drawings:

[0072] Figure 1 The automatic driving obstacle intention prediction and avoidance method provided by the application is shown in the whole step schematic block diagram.

[0073] Figure 2 The intention prediction and avoidance process of the automatic driving obstacle intention prediction and avoidance method provided by the application is shown in the schematic block diagram. DETAILED DESCRIPTION

[0074] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The preferred embodiments of the present application will be described in detail with reference to the drawings.

[0075] The application discloses an automatic driving obstacle intention prediction and avoidance method, as shown in Figure 1 and Figure 2 , comprising:

[0076] S1, real-time acquisition of image data and three-dimensional point cloud data of the front obstacle in the automatic driving process;

[0077] S2, extracting static features of the obstacle from the image data and dynamic features of the obstacle from the three-dimensional point cloud data;

[0078] S3, weighting the static features and dynamic features respectively by using an attention mechanism, and fusing the weighted static features and dynamic features to obtain a fusion feature vector;

[0079] S4, inputting the fusion feature vector into a plurality of independently trained deep learning models, weighting and fusing the output results of the plurality of deep learning models based on the model differences of the plurality of deep learning models to obtain the predicted intention of the obstacle;

[0080] S5, setting the association constraint of the vehicle running state and the input control according to the predicted intention, and establishing an avoidance strategy model with path length and time cost as double optimization objectives;

[0081] S6, continuously updating the input control within the prediction time domain until the double optimization objective value of the avoidance strategy model is iterated to the minimum within the association constraint, and outputting the time sequence of the input control to form the avoidance strategy;

[0082] S7, acquiring the real-time running state of the vehicle in the automatic driving process, combining the avoidance strategy and the real-time running state to generate the final vehicle execution instruction.

[0083] Through the implementation of the above-mentioned automatic driving obstacle intention prediction and avoidance method embodiment, real-time image data and three-dimensional point cloud data are acquired and processed in parallel, the static characteristics and dynamic characteristics of the obstacle are simultaneously extracted, and the limitation of single data source analysis is broken through. The attention mechanism is used to weight the two types of characteristics respectively, the weight distribution is dynamically adjusted according to the importance of the characteristics, then a high-dimensional fusion feature vector with complementary information is formed by fusion, and the feature expression capability is significantly improved. The fusion feature vector is input into multiple independently trained deep learning models for multi-modal analysis, the diversity output results generated by the structural differences of the models are integrated through a weighted fusion mechanism to integrate the prediction advantages of multiple models, effectively overcome the deviation of a single model, and obtain a high-precision predicted intention of the future motion state and behavior motivation of the obstacle. Based on the predicted intention, the associated constraint conditions of the vehicle operating state and the input control are set, a mathematical optimization model with path length and time cost as dual optimization objectives is constructed, the input control is continuously updated within the prediction time domain, and the dual optimization objective values are iteratively converged to the minimum values under the condition of meeting all associated constraint conditions, and finally an avoidance strategy composed of the input control sequence after optimization is output. Thus, the final execution instruction is dynamically generated in real time according to the actual operating state of the vehicle, so as to solve the obstacle intention misjudgment problem and the avoidance strategy rigidification problem caused by the limitation of static feature recognition and the absence of dynamic behavior prediction in the prior art. That is, the embodiment of the present application realizes full-dimensional accurate perception of obstacle intention, probabilistic modeling and prediction of motion state, and real-time self-adaptive optimization of avoidance decision in a complex environment through a multi-level feature fusion mechanism, a multi-model collaborative prediction mechanism and a dynamic optimization solving mechanism, so that the automatic driving system has a forward-looking disposal capability for sudden conditions, while ensuring the unity of the optimality and timeliness of the driving trajectory. Not only can the false triggering rate of emergency avoidance be greatly reduced, but also the smoothness and traffic flow integration degree of continuous avoidance operation can be improved, thereby breaking through the technical bottleneck of the traditional system based on static characteristics and preset rules, improving the driving safety redundancy, and significantly optimizing the road traffic efficiency.

[0084] Further, the image data and three-dimensional point cloud data of the front obstacle in the automatic driving process are acquired, including:

[0085] The image data and three-dimensional point cloud data of the front obstacle in the automatic driving process are acquired in real time, the image data contains appearance information of the obstacle, and the three-dimensional point cloud data contains shape and position information of the obstacle;

[0086] The acquired image data is preprocessed, including removing noise from the image data by wavelet transform, and enhancing the contrast of the image data by histogram equalization, and the function expression of the image data preprocessing is:

[0087]

[0088] In the formula, represents pre-processed image data, represents pre-processed image data, represents a composite transform function of noise removal and contrast enhancement, represents residual noise.

[0089] Through the implementation of the above-mentioned automatic driving obstacle intention prediction and avoidance method embodiment, the pre-processed image data significantly improves the noise resistance and texture clarity while retaining the integrity of the appearance information. The noise removal operation can suppress invalid pixel disturbances caused by environmental interference, and the contrast enhancement can enhance the target edge and detail features. The existence of residual noise avoids the loss of features caused by excessive smoothing, so that the three-dimensional point cloud data can maintain the original state transmission to fully maintain the spatial precision and real-time of its shape and position information. The high-quality image data after preprocessing provides a low-noise and high-discrimination input basis for static feature extraction, which can overcome the feature distortion problem caused by single data quality degradation in complex scenes such as rain, fog, and night, and provide high-fidelity data support for the construction of fused feature vectors, thereby improving the perception robustness of the automatic driving system to the essential attributes (appearance / shape / position) of obstacles in harsh environments.

[0090] As mentioned above, preferably, during the driving of the autonomous vehicle, the image data and three-dimensional point cloud data of the front obstacles are collected in real time by sensors such as high-precision cameras (resolution of 4096x3072 pixels, frame rate of 60 fps) and laser radars (LiDAR, scanning frequency of 20 Hz, point cloud density of 2 million points per second). The image data records the appearance information of the obstacles in detail, such as posture (standing, walking, running, etc.), clothing color (red, blue, green, etc.), size (height, width), etc.; the three-dimensional point cloud data accurately depicts the shape (such as human body contour, luggage shape) and position information (distance from the vehicle, direction angle, etc.) of the obstacles. In order to improve the accuracy of subsequent processing, the collected data is pre-processed: high-frequency noise in the image data is removed using wavelet transform (Daubechies 4 wavelet basis, 5 layers of decomposition), and the contrast of the image is enhanced using histogram equalization (contrast stretching to 0-255 range).

[0091] Further, the static features of the obstacles are extracted from the image data, including:

[0092] The appearance flow network constructed by multiple convolutional layers extracts the static features of the obstacles from the pre-processed image data, and the static features include the shape, texture, and color histogram of the obstacles. The function expression of the appearance flow network for extracting static features is:

[0093]

[0094] In the formula, represents the feature map output by the appearance flow network, represents a nonlinear activation function, represents a weight matrix of a convolution kernel, represents image data input to the appearance flow network, represents a convolution bias term.

[0095] Through implementation of the above-mentioned embodiment of the automatic driving obstacle intention prediction and avoidance method, the appearance flow network learns deep features of the spatial dimension of the input image data through the convolution kernel weight matrix, and the high-order nonlinear modeling capability of the geometric structure and surface properties of the obstacle is strengthened in combination with the nonlinear activation function. The introduction of the convolution bias term can effectively adapt the feature distribution offset of the image data under different brightness conditions. The multi-level convolution structure of the network retains the integrity of the macro shape contour while extracting microscopic texture details, and the output feature map has both local feature resolution and global structure consistency, significantly enhancing the robustness of the characterization of the nature of the obstacle under complex scenes such as rain and fog obstruction and night low illumination, and providing high-quality static feature input for the intention prediction.

[0096] As described above, preferably, the appearance flow network adopts a deep convolutional neural network, specifically a ResNet-101 model (Deep Residual Network with 101 layers), for feature extraction. The ResNet-101 contains 101 convolutional layers, and the last fully connected layer is removed to output feature maps. These feature maps contain static features such as the shape, texture, and color histogram of the obstacle. Among them, the nonlinear activation function adopts a ReLU nonlinear activation function (Rectified Linear Unit).

[0097] Further, dynamic features of the obstacle are extracted from the three-dimensional point cloud data, including:

[0098] The motion flow network constructed by the gating cycle extracts dynamic features of the obstacle from the three-dimensional point cloud data. The dynamic features include the speed, direction, and acceleration of the obstacle. The function expression of the motion flow network for extracting dynamic features is:

[0099]

[0100]

[0101]

[0102] In the formula, represents a reset gate, reset gate, reset gate bias, update gate, update gate weight, update gate bias, candidate hidden state, candidate hidden state weight, candidate hidden state bias, non-linear activation function, previous time step hidden state, current time step input three-dimensional point cloud data, hyperbolic tangent activation function, element-wise multiplication.

[0103] Through the implementation of the above-mentioned automatic driving obstacle intention prediction and avoidance method embodiment, the motion flow network constructed by the gating cycle extracts the dynamic characteristics (including speed, direction, acceleration) of the obstacle from the three-dimensional point cloud data, and accurately defines the extraction process of the dynamic characteristics in the function expression: the reset gate calculates the non-linear combination of the previous time step hidden state and the current input through the weight and the bias term , the update gate calculates the combination of the two through the weight and the bias term , the candidate hidden state , then based on the reset gate , the weighted result of the previous time step hidden state and the current input is calculated through the weight and the bias term , and the hyperbolic tangent activation output is calculated, so that through the double gating mechanism of the reset gate and the update gate, the effective information of the dynamic characteristics of the obstacle is adaptively filtered and the redundant interference is suppressed, the candidate hidden state is used to accurately capture the continuous evolution law of the motion characteristics of the obstacle by modeling the spatio-temporal correlation of the real-time three-dimensional point cloud data, so that the dynamic characteristic extraction process can deeply integrate the time sequence behavior pattern and the instantaneous state change of the obstacle, significantly enhance the representation robustness of the dynamic evolution of speed, direction and acceleration in complex motion trajectories (such as accelerating lane changing, sharp turning), and provide high-precision dynamic characteristics with strong time sequence dependence for intention prediction.

[0104] As described above, preferably, the motion flow network adopts a Gated Recurrent Unit (GRU) to convert the point cloud data collected by the lidar into time series data and input into the motion flow network to extract the dynamic characteristics of the obstacles. Through the motion flow network, the motion states of the obstacles such as speed, direction, acceleration, etc. and their trends over time can be captured.

[0105] Further, the automatic driving obstacle intention prediction and avoidance method further comprises a method of fusing static features and dynamic features, comprising:

[0106] A first weight matrix and a first bias term defining an attention mechanism are defined, the static features are mapped into a calculation space of attention weights by using the defined first weight matrix, and the attention weights of the static features are calculated by a normalization function, and the calculation function expression of the static feature attention weights is:

[0107]

[0108] A second weight matrix and a second bias term defining an attention mechanism are defined, the dynamic features are mapped into a calculation space of attention weights by using the defined second weight matrix, and the attention weights of the dynamic features are calculated by a normalization function, and the calculation function expression of the dynamic feature attention weights is:

[0109]

[0110] In the formula, indicates the attention weights of the static features, indicates the first weight matrix, indicates the static features, indicates the first bias term, indicates the attention weights of the dynamic features, indicates the second weight matrix, indicates the dynamic features, indicates the second bias term, indicates the normalization function;

[0111] The weighted static features and dynamic features are spliced to obtain a fusion feature vector, and the function expression of the splicing of the static features and the dynamic features is:

[0112]

[0113] In the formula, indicates the spliced fusion feature vector, indicates element multiplication.

[0114] By implementing the above automatic driving obstacle intention prediction and avoidance method embodiment, a two-parameter space is constructed using a separate first weight matrix and a second weight matrix, so that static features and dynamic features are independently nonlinearly mapped in the calculation process of attention weights, and then attention weights strictly matched with feature importance are respectively generated by a normalization function. Subsequently, element multiplication is used to perform fine fusion of feature dimension alignment on the weighted static features and dynamic features, which can break through the feature expression confusion problem caused by weight sharing in traditional feature splicing, significantly enhancing the collaborative representation ability of the fusion feature vector for the multi-modal attributes of obstacles in complex scenes. That is, when environmental interference weakens the reliability of static features, the first weight matrix automatically reduces the weight allocation of blurred textures, while the second weight matrix synchronously enhances the contribution proportion of dynamic features, thereby maintaining the information completeness of the fusion feature vector in complex environments, providing a robust input basis for adaptive environmental changes for subsequent obstacle intention prediction.

[0115] As described above, preferably, the normalization function adopts a Softmax function (Soft Maximum Function).

[0116] For further illustration, a task of detecting an obstacle in front of an autonomous vehicle is processed:

[0117] The appearance flow network is responsible for extracting static features of the obstacle from a single image frame, such as shape, color, texture, etc., which are represented as a vector of dimension 128 ; the motion flow network is responsible for extracting dynamic features of the obstacle from consecutive video frames, such as speed, acceleration, direction change, etc., which are also represented as a vector of dimension 128 .

[0118] In order to fuse the two feature vectors into a fusion feature vector, an attention mechanism is introduced. First, two weight matrices and are defined, both of which have a dimension of 1x128, for mapping appearance features and motion features to the calculation space of attention weights. At the same time, two bias terms and are defined, both of which have a dimension of 1, for adjusting the calculation of attention weights.

[0119] Then, the attention weights and of static features and dynamic features are calculated using the softmax function;

[0120] Suppose

[0121] = [0.6, 0.2, 0.1, ..., 0.1] (Only some elements are shown; it is actually a 128-dimensional vector).

[0122] = [0.5, 0.3, 0.1, ..., 0.1] (Only some elements are shown).

[0123] Attention weight and Indicating static features during the fusion process and dynamic features The importance of each element.

[0124] Then, the weighted features are concatenated to obtain a fusion feature vector with a dimension of 256, which contains fusion information of appearance features and motion features, providing richer feature representations for subsequent obstacle classification and detection tasks.

[0125] Furthermore, the autonomous driving obstacle intent prediction and avoidance method also includes methods that utilize multiple deep learning models to predict the fused feature vector to obtain the predicted intent, including:

[0126] The fused feature vectors of multiple known intentional obstacles during historical autonomous driving processes are obtained and used as training data.

[0127] Multiple deep learning models, each constructed from convolutional neural networks, fully connected layers, and nonlinear activation functions, are used and trained independently with different initialization parameters and training data.

[0128] The fused feature vectors of obstacles acquired in real time are input into multiple trained deep learning models, and the weights of the output results of each deep learning model are calculated using the random forest algorithm.

[0129] The output of each deep learning model is multiplied by its corresponding weight, and the weighted outputs of all deep learning models are summed and fused to obtain the final prediction intent. The function expression for the weighted fusion of multiple deep learning models is as follows:

[0130]

[0131] In the formula, Indicates the final predictive intent. This represents the output of the i-th deep learning model. This represents the weight of the i-th deep learning model.

[0132] By implementing the above automatic driving obstacle intention prediction and avoidance method embodiment, the weight of each deep learning model output result is dynamically calculated using the random forest algorithm. Due to the differences in initialization parameters and training data, each independently trained deep learning model forms inherent heterogeneity in the model structure, making the prediction results of different deep learning models on the same fusion feature vector have significant diversity. Based on the fusion feature vector training data of known intention obstacles in the historical automatic driving process, the random forest algorithm adaptively assigns the weight of each deep learning model through feature importance analysis, ensuring that the decision contribution of highly adaptive deep learning models is automatically strengthened when the obstacle motion pattern mutates. Finally, through the weighted fusion mechanism of multiplying the weight and the output result of the corresponding deep learning model and then adding them all together, the generalization bottleneck and scene dependency limitation of a single deep learning model are broken through, making the predicted intention result have both the multi-model redundancy advantage and the random forest dynamic calibration capability, significantly improving the recognition robustness and prediction fault tolerance of the real intention of obstacles in complex interactive scenarios (such as multiple obstacles parallel lane changing and intention ambiguous deceleration behavior).

[0133] As mentioned above, preferably, the nonlinear activation function of the deep learning model adopts a softmax activation function (SoftMaximum Activation Function). In order to improve the accuracy of classification, an ensemble learning method such as random forest or gradient boosting tree is used to fuse the output results of multiple deep learning models.

[0134] Further illustration:

[0135] Suppose there is an automatic driving scene containing 10 obstacles, each with 256-dimensional comprehensive features. If three deep learning models (e.g. Model A, Model B and Model C) are trained to predict the intentions of these obstacles. These models have the same network structure but use different initialization parameters and training data.

[0136] During training, cross-validation can be used to evaluate the performance of each model and select the optimal model parameters. Suppose the accuracy of Model A, Model B and Model C on the validation set is 90%, 85% and 88% respectively.

[0137] Then, a random forest is used as an ensemble learning method to calculate the weight of each model. Random forest improves the accuracy of classification by constructing multiple decision trees and integrating their prediction results. In the embodiment of the invention, 50 decision trees can be constructed to calculate the weight of each model.

[0138] Through the calculation of the random forest, the weights of Model A, Model B and Model C are 0.4, 0.3 and 0.3 respectively. These weights reflect the importance of each model in the final prediction.

[0139] Finally, the output results of each model are multiplied by their corresponding weights, and the weighted output results are added to obtain the final predicted intention. This predicted intention represents the probability distribution of each obstacle intention and can be used for decision-making and control of the autonomous driving system.

[0140] Further, the association constraints of the vehicle operating state and input control are set according to the predicted intention, and an avoidance strategy model is established with path length and time cost as dual optimization objectives, including:

[0141] According to the dynamics characteristics of the vehicle and the road constraints, the association of the operating state and input control in the vehicle autonomous driving process is set, and the functional expression of the operating state and input control association is:

[0142]

[0143] wherein, represents the vehicle operating state at time t, represents the vehicle input control at time t, represents the system dynamics model of the vehicle;

[0144] According to the predicted intention of the obstacle, the constraints of the operating state and input control in the vehicle autonomous driving process are set, and the functional expression of the operating state and input control constraints is:

[0145]

[0146] wherein, represents a set of constraint conditions, represents the predicted intention of the obstacle at time t;

[0147] An avoidance strategy model in the prediction time domain is established with path length and time cost as dual optimization objectives, and the functional expression of the avoidance strategy model is:

[0148]

[0149] wherein, represents the dual optimization objective value output by the avoidance strategy model, and represents a weight coefficient, and T represents the prediction time domain.

[0150] By implementing the above-mentioned automatic driving obstacle intention prediction and avoidance method embodiment, the vehicle's own physical characteristics (such as steering inertia, driving response) and road geometric constraints (such as curvature boundary, slope limit) are accurately embedded using a system dynamics model, so that the running state update process strictly follows the actual mechanical dynamics law. At the same time, the time-varying obstacle prediction intention is taken as the adaptive input of the constraint function, forcing the avoidance strategy to respond to external threat evolution in the prediction time domain, and realizing the rolling joint optimization of path length and time cost in the prediction time domain through a double optimization objective function. Among them, represents the vehicle running state at time t, such as vehicle position, speed, acceleration, etc. represents the vehicle input control at time t, such as throttle, brake, steering, etc. represents the system dynamics model of the vehicle, taking into account the dynamics characteristics of the vehicle and the road constraints; g is a set of constraint conditions to ensure that the avoidance strategy does not violate traffic rules, avoid collisions, maintain lanes, etc. These constraint conditions may include speed limits, steering angle limits, collision avoidance with other vehicles or obstacles, etc.; the weight coefficient and are used to balance the priority between different optimization objectives, which cooperatively adjust to break through the barrier that efficiency and safety cannot be considered in traditional single-objective optimization, and finally under the double closed-loop action of state transfer constraints and intention driven constraints, the continuously updated input control sequence converges systematically to the dynamic trajectory solution that minimizes the double optimization objective value, that is, by iteratively solving the double optimization objective problem, that is, under the premise of meeting the constraint conditions, finding the input control sequence that minimizes the avoidance strategy model J , so as to obtain a series of optimal input controls, which will constitute the avoidance strategy to guide the automatic avoidance behavior of the vehicle.

[0151] Based on the above, in detail: in the prediction time domain, the input control is executed in a closed-loop rolling optimization manner, so that the double optimization objective value of the avoidance strategy model is continuously iteratively updated under the double boundary restrictions of the system dynamics model and the intention constraint set . Each iteration re-solves the constraint optimization problem according to the current vehicle running state , and adaptively adjusts the spatio-temporal distribution of the input control by gradient descent method until the double optimization objective value converges to the minimum value in the current prediction time domain. This mechanism breaks through the static limitations of traditional open-loop decision-making, so that the time sequence of the final output input control strictly obeys the mechanical constraints of the vehicle's steering inertia and road curvature, the safety avoidance boundary of the dynamically coupled obstacle prediction intention, and the path length and time cost in the weight coefficients and The precise allocation achieves global optimal balance, thereby generating a physically feasible, agile, and energy-efficient avoidance strategy at the output end.

[0152] Furthermore, the real-time operating status of the vehicle during autonomous driving is acquired, and the final vehicle execution commands are generated by combining the avoidance strategy and the real-time operating status, including:

[0153] The desired operating state of the vehicle is determined based on the avoidance strategy, and the current actual operating state of the vehicle during the autonomous driving process is collected.

[0154] The target acceleration for vehicle control execution is calculated based on the desired and actual operating states. The expression for the calculation function of the target acceleration is as follows:

[0155]

[0156] In the formula, This indicates the target acceleration for vehicle control execution. Indicates the desired speed. Indicates the expected acceleration. This indicates the vehicle's current actual speed. This indicates the vehicle's current actual acceleration. This is the proportional gain coefficient. The differential gain coefficient;

[0157] Based on the calculated target acceleration, output the corresponding vehicle control commands.

[0158] Through the implementation of the above-described autonomous driving obstacle intent prediction and avoidance method embodiment, the proportional gain coefficient is utilized. For the desired speed Compared with actual speed The difference is dynamically scaled and adjusted, while utilizing the differential gain coefficient. For desired acceleration With actual acceleration The deviation is corrected and compensated in real time, forming a dual closed-loop negative feedback mechanism for both velocity and acceleration. This feedforward-feedback composite control strategy, which integrates the actual operating state, can overcome the limitations of traditional open-loop control that relies solely on the desired operating state, enabling the target acceleration to be... The calculation results can instantly respond to the transient error between the vehicle's motion state and the avoidance strategy requirements, that is, when the actual acceleration... Due to sudden changes in road surface adhesion or mechanical delay lagging behind the desired acceleration When, the differential term This will result in compensatory incremental forced closed-loop convergence; when the actual speed Deviation from expected speed When, proportional term The synergy of the two makes the finally generated vehicle execution instruction have both dynamic deviation correction capability and motion continuity guarantee, completely solves the problems of acceleration jerk, speed oscillation and response delay caused by the lack of state feedback in traditional methods while meeting the safety constraints of the avoidance strategy, and significantly improves the smoothness and working condition adaptability of automatic driving control.

[0159] As described above, the avoidance strategy is delivered to the control module of the automatic driving system, which realizes the automatic avoidance of the vehicle by controlling the throttle, brake, steering and other actuators of the vehicle. When executing the avoidance strategy, the control module will calculate the corresponding control instructions according to the target acceleration such as adjusting the throttle opening, brake force or steering angle. These control instructions are implemented through the throttle, brake, steering and other actuators of the vehicle to realize precise control of the vehicle, so that the autonomous vehicle can accurately track the desired acceleration and speed to realize safe and stable avoidance operation.

[0160] The application also discloses a prediction and avoidance system adopting the automatic driving obstacle intention prediction and avoidance method, and the system comprises:

[0161] A data acquisition module is configured to acquire image data and three-dimensional point cloud data of a front obstacle in real time during automatic driving;

[0162] A feature extraction module is configured to extract static features of the obstacle from the image data and dynamic features of the obstacle from the three-dimensional point cloud data;

[0163] A feature fusion module is configured to weight the static features and the dynamic features respectively by using an attention mechanism, and fuse the weighted static features and dynamic features to obtain a fusion feature vector;

[0164] An intention prediction module is configured to input the fusion feature vector into a plurality of independently trained deep learning models, weight and fuse the output results of the plurality of deep learning models based on the model differences of the plurality of deep learning models to obtain a predicted intention of the obstacle;

[0165] A model establishment module is configured to set an associated constraint between a vehicle operating state and an input control according to the predicted intention, and establish an avoidance strategy model with path length and time cost as dual optimization objectives;

[0166] An avoidance strategy generation module is configured to continuously update the input control within a prediction time domain until the dual optimization objective value of the avoidance strategy model is iteratively minimized within the associated constraint, and output a time sequence of the input control to form the avoidance strategy;

[0167] The control instruction generation module is used for acquiring a real-time running state of the vehicle in the automatic driving process, and generating a final vehicle execution instruction in combination with an avoidance strategy and the real-time running state.

[0168] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the automatic driving obstacle intention prediction and avoidance method.

[0169] The application further discloses a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the automatic driving obstacle intention prediction and avoidance method.

[0170] The application is described according to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the specific embodiments. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. The computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0171] The computer program instructions can also be stored in a computer readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product comprising instruction apparatus, which realizes the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0172] The computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0173] It should be understood that the above examples are only used to illustrate the technical solutions of the present application, but not limit the technical solutions of the present application. Those skilled in the art can modify the technical solutions described in the above examples, or make equivalent replacements to some of the technical features. All these modifications and replacements shall belong to the protection scope of the present application.

Claims

1. An automatic driving obstacle intention prediction and avoidance method, characterized in that, The method comprises the following steps: real-time acquisition of image data and three-dimensional point cloud data of the front obstacle in the automatic driving process; extracting static features of the obstacle from the image data and dynamic features of the obstacle from the three-dimensional point cloud data; using attention mechanism to weight the static features and the dynamic features respectively, and fusing the weighted static features and dynamic features to obtain a fusion feature vector; inputting the fusion feature vector into a plurality of independently trained deep learning models, weighting and fusing the output results of the plurality of deep learning models based on the model differences of the plurality of deep learning models to obtain the predicted intention of the obstacle; setting the association constraint of the vehicle operating state and the input control according to the predicted intention, and establishing an avoidance strategy model with path length and time cost as double optimization objectives; continuously updating the input control in the prediction time domain until the double optimization objective value of the avoidance strategy model is iterated to the minimum within the association constraint, and outputting the time sequence of the input control to form an avoidance strategy; acquiring the real-time operating state of the vehicle in the automatic driving process, combining the avoidance strategy and the real-time operating state to generate a final vehicle execution instruction.

2. The method of claim 1, wherein, The method comprises the following steps: real-time acquisition of image data and three-dimensional point cloud data of the front obstacle in the automatic driving process; preprocessing the acquired image data, including removing noise from the image data using wavelet transform, and enhancing the contrast of the image data using histogram equalization, the function expression of the image data preprocessing being: wherein denotes the pre-processed image data, denotes the pre-processed image data, denotes a composite transform function of noise removal and contrast enhancement, denotes residual noise.

3. The method of claim 2, wherein, extracting static features of the obstacle from the image data, including: extracting static features of the obstacle from the preprocessed image data using an appearance flow network constructed by a plurality of convolution layers, the static features including shape, texture, and color histogram of the obstacle, the function expression of the appearance flow network for extracting static features being: wherein represents a feature map output by the appearance flow network, represents a non-linear activation function, represents a weight matrix of a convolution kernel, represents image data input to the appearance flow network, represents a convolution bias term.

4. The method of claim 1, wherein, extracting dynamic features of the obstacle from the three-dimensional point cloud data, including: extracting dynamic features of the obstacle from the three-dimensional point cloud data using a motion flow network constructed by a gated recurrent unit, the dynamic features including velocity, direction, and acceleration of the obstacle, the function expression of the motion flow network for extracting dynamic features being: wherein, denotes a reset gate, denotes a reset gate weight, denotes a reset gate bias term, denotes an update gate, denotes an update gate weight, denotes an update gate bias term, denotes a candidate hidden state, denotes a candidate hidden state weight, denotes a candidate hidden state bias term, denotes a nonlinear activation function, denotes a previous time step hidden state, denotes a three-dimensional point cloud data of a current time step input, denotes a hyperbolic tangent activation function, denotes an element-wise multiplication.

5. The method of claim 1, wherein, The automatic driving obstacle intention prediction and avoidance method further comprises a fusion method of the static features and the dynamic features, comprising: defining a first weight matrix and a first bias term of the attention mechanism, mapping the static features to a calculation space of attention weights using the defined first weight matrix, and calculating the attention weights of the static features through a normalization function, the calculation function expression of the static feature attention weights being: The second weight matrix and the second bias term defining the attention mechanism are defined, the dynamic features are mapped into the calculation space of the attention weight by using the defined second weight matrix, and the attention weight of the dynamic features is calculated by a normalization function, and the calculation function expression of the dynamic feature attention weight is: wherein, denotes an attention weight for a static feature, denotes a first weight matrix, denotes a static feature, denotes a first bias term, denotes an attention weight for a dynamic feature, denotes a second weight matrix, denotes a dynamic feature, denotes a second bias term, denotes a normalization function; The weighted static features and dynamic features are spliced to obtain the fusion feature vector, and the function expression of the static features and dynamic features splicing is: In the formula, represents the fused feature vector after concatenation, represents element-wise multiplication.

6. The method of claim 1, wherein, The automatic driving obstacle intention prediction and avoidance method further comprises a method for predicting the fusion feature vector of the obstacle to obtain the predicted intention by using a plurality of deep learning models, comprising: Obtaining the fusion feature vector of a plurality of known intention obstacles in the historical automatic driving process and using it as training data; A plurality of deep learning models are independently trained using different initialization parameters and training data, wherein each deep learning model is constructed by a convolutional neural network, a fully connected layer and a nonlinear activation function; The fusion feature vector of the obstacle obtained in real time is input into a plurality of trained deep learning models, and the weight of the output result of each deep learning model is calculated by using a random forest algorithm; The output result of each deep learning model is multiplied by the corresponding weight, and the weighted output results of all deep learning models are added and fused to obtain the final predicted intention, and the function expression of the weighted fusion of a plurality of deep learning models is: In the formula, represents the final prediction intention, represents the output result of the i-th deep learning model, represents the weight of the i-th deep learning model.

7. The method of claim 1, wherein, The associated constraints of the vehicle operating state and the input control are set according to the predicted intention, and an avoidance strategy model is established with path length and time cost as dual optimization objectives, comprising: The association of the operating state and the input control in the automatic driving process of the vehicle is set according to the dynamic characteristics of the vehicle and the road constraints, and the function expression of the association of the operating state and the input control is: In the formula, denotes the vehicle operating state at time t, denotes the vehicle input control at time t, denotes a system dynamics model of the vehicle; The constraints of the operating state and the input control in the automatic driving process of the vehicle are set according to the predicted intention of the obstacle, and the function expression of the constraints of the operating state and the input control is: In the formula, denotes a set of constraints, denotes the obstacle prediction intention at time t; An avoidance strategy model in the prediction time domain is established with path length and time cost as dual optimization objectives, and the function expression of the avoidance strategy model is: In the formula, denotes the double optimization target value output by the avoidance strategy model, and denotes a weight coefficient, and T denotes a prediction time domain.

8. The method of claim 1, wherein, The real-time operating state of the vehicle in the automatic driving process is obtained, and the final vehicle execution instruction is generated by combining the avoidance strategy and the real-time operating state, comprising: The desired operating state of the vehicle is determined according to the avoidance strategy, and the current actual operating state of the vehicle in the automatic driving process is acquired; The target acceleration of the vehicle control execution is calculated according to the desired operating state and the actual operating state, and the calculation function expression of the target acceleration is: wherein denotes a target acceleration of the vehicle control execution, denotes a desired speed, denotes a desired acceleration, denotes a current actual speed of the vehicle, denotes a current actual acceleration of the vehicle, is a proportional gain factor, is a differential gain factor; The corresponding vehicle control instruction is output according to the calculated target acceleration.

9. A prediction and avoidance system employing the automated driving obstacle intent prediction and avoidance method of any one of claims 1-8, wherein, The system comprises: A data acquisition module for acquiring image data and three-dimensional point cloud data of a front obstacle in an automatic driving process in real time; A feature extraction module for extracting static features of the obstacle from the image data and dynamic features of the obstacle from the three-dimensional point cloud data; a feature fusion module configured to weight the static feature and the dynamic feature respectively by using an attention mechanism, and to fuse the weighted static feature and the dynamic feature to obtain a fusion feature vector; an intention prediction module configured to input the fusion feature vector into a plurality of independently trained deep learning models, to weight and fuse output results of the plurality of deep learning models based on model differences of the plurality of deep learning models, and to obtain a predicted intention of the obstacle; a model establishment module configured to set an associated constraint of a vehicle operating state and an input control according to the predicted intention, and to establish an avoidance strategy model with path length and time cost as dual optimization objectives; an avoidance strategy generation module configured to continuously update the input control within a prediction time domain until the dual optimization objective value of the avoidance strategy model is iterated to a minimum within the associated constraint, and to output a time sequence of the input control to form an avoidance strategy; a control instruction generation module configured to obtain the operating state of the vehicle in real time during automatic driving, and to generate a final vehicle execution instruction in combination with the avoidance strategy and the real-time operating state.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the automatic driving obstacle intention prediction and avoidance method of any one of claims 1-8.

Images