Automatic driving control method, device and equipment based on potential world model guidance and storage medium

Abstract

The invention discloses an automatic driving control method, device and equipment based on potential world model guidance and a storage medium, and relates to the technical field of automatic driving, and the automatic driving control method based on potential world model guidance comprises the steps: obtaining sensor data, vehicle image data and vehicle navigation information; performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data; performing feature extraction based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map; and obtaining predicted trajectory data based on the vehicle planning data, the current frame feature map and the target future frame feature map, so that the vehicle completes automatic driving control according to the predicted trajectory data. According to the invention, the planning accuracy of the automatic driving vehicle in the dynamic traffic environment is improved, and the prediction capability of future environment change is enhanced, so that the safety of an automatic driving system is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving control method, apparatus, device and storage medium guided by a potential world model. Background Art

[0002] With the development of autonomous driving technology, higher requirements are placed on the decision-making and planning capabilities of vehicles. In complex traffic environments, vehicles need to be able to accurately predict and respond to future changes in the environment to ensure driving safety and efficiency.

[0003] At present, most end-to-end planning methods in the field of autonomous driving rely on data from the current frame to make decisions, and lack the ability to predict future environmental states. These methods usually only consider immediate sensor input and vehicle status, and do not fully utilize historical data and environmental information to predict future changes. This method may perform well in static or slowly changing environments, but in dynamic traffic environments, due to the lack of foresight of upcoming events, it is often unable to make timely and accurate responses.

[0004] Existing technologies have poor adaptability in dynamic traffic environments and cannot respond to traffic changes ahead in a timely manner, such as vehicle lane changes, pedestrian crossings, etc. In addition, since the future environmental state cannot be predicted, these planning methods may reduce the safety and reliability of the autonomous driving system, and may not provide sufficient warnings and response measures in emergency situations, increasing the risk of accidents. Therefore, how to improve the autonomous driving system's ability to predict future environmental changes has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of this application is to provide an autonomous driving control method, device, equipment and storage medium based on the guidance of a potential world model, aiming to solve the technical problem of how to improve the autonomous driving system's ability to predict future environmental changes.

[0006] To achieve the above objectives, the present application proposes an automatic driving control method based on guidance of a potential world model, the method comprising:

[0007] Acquire sensor data, vehicle image data, and vehicle navigation information;

[0008] Performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data;

[0009] Perform feature extraction based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map;

[0010] Based on the vehicle planning data, the current frame feature map and the target future frame feature map, predicted trajectory data is obtained so that the vehicle can complete automatic driving control according to the predicted trajectory data.

[0011] In one embodiment, data fusion is performed based on the sensor data and the vehicle navigation information to obtain vehicle planning data, including:

[0012] Obtaining position data, speed data, heading angle data and vehicle size data based on the sensor data;

[0013] The position data, the speed data, the heading angle data and the vehicle size data are stacked, and data encoding is performed through a multi-layer perceptron structure to obtain vehicle history data;

[0014] Data fusion is performed based on the vehicle historical data and the vehicle navigation information to obtain vehicle planning data.

[0015] In one embodiment, data fusion is performed based on the vehicle historical data and the vehicle navigation information to obtain vehicle planning data, including:

[0016] The vehicle navigation data is encoded by a multi-layer perceptron structure to obtain the vehicle navigation data;

[0017] Calculate the vehicle navigation data and the vehicle history data by a neural network algorithm to obtain a weight vector and a scaling vector;

[0018] Vehicle planning data is obtained based on the weight vector, the scaling vector, the vehicle history data, and the vehicle navigation data.

[0019] In one embodiment, feature extraction is performed based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map, including:

[0020] Perform feature extraction based on the vehicle image data and a preset world model to obtain a historical frame feature map;

[0021] Recording vehicle trajectory data through a network training method to obtain a vehicle trajectory feature graph, wherein the vehicle trajectory data is obtained based on the sensor;

[0022] The vehicle trajectory feature map and the historical frame feature map are spliced ​​in time sequence to obtain a current frame feature map;

[0023] The current frame feature map is decoded to obtain a target future frame feature map.

[0024] In one embodiment, decoding the current frame feature map to obtain a target future frame feature map includes:

[0025] Based on the dimension of the current feature map, obtaining a preset noise value;

[0026] Performing convolution encoding on the current frame feature map to obtain a reference current frame feature map;

[0027] Performing feature data constraints on the preset noise value to obtain a noise weight value;

[0028] Obtaining an initial future frame feature map by combining the reference current frame feature map, the noise weight value and the preset noise value through a gated cycle unit;

[0029] The initial future frame feature map is adjusted to obtain a target future frame feature map.

[0030] In one embodiment, adjusting the initial future frame feature map to obtain a target future frame feature map includes:

[0031] Obtaining current frame laser radar data based on the sensor data;

[0032] Performing voxel feature encoding on the current frame laser radar data to obtain a future frame laser radar feature map;

[0033] Obtaining a feature loss value through the future frame laser radar feature map and the future frame feature map;

[0034] The initial future frame feature map is adjusted according to the feature loss value to obtain a target future frame feature map.

[0035] In one embodiment, obtaining predicted trajectory data based on the vehicle planning data, the current frame feature map, and the target future frame feature map includes:

[0036] Get the preset transformer model;

[0037] A cross attention mechanism is introduced based on the preset transformer model, and predicted trajectory data is obtained through the vehicle planning data, the current frame feature map and the target future frame feature map.

[0038] In addition, to achieve the above purpose, the present application also proposes an automatic driving control device based on potential world model guidance, the device comprising:

[0039] An acquisition module, used to acquire sensor data, vehicle image data and vehicle navigation information;

[0040] An obtaining module, used for performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data;

[0041] A fusion module, used for extracting features based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map;

[0042] The completion module is used to obtain predicted trajectory data based on the vehicle planning data, the current frame feature map and the future frame feature map, so that the vehicle can complete the automatic driving control according to the predicted trajectory data.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes an autonomous driving control device based on the guidance of a potential world model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the autonomous driving control method based on the guidance of a potential world model as described above.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the autonomous driving control method based on the guidance of the potential world model as described above are implemented.

[0045] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the autonomous driving control method based on the guidance of the potential world model as described above.

[0046] One or more technical solutions proposed in this application have at least the following technical effects:

[0047] This application first provides comprehensive environmental perception and vehicle status information for the autonomous driving system by collecting sensor data, vehicle image data and vehicle navigation information, which is the basis for accurate decision-making and planning. Secondly, sensor data and vehicle navigation information are used for data fusion to generate vehicle planning data, which helps to integrate the historical and real-time status of the vehicle, as well as the predetermined navigation path, and provides rich contextual information for subsequent feature extraction and trajectory prediction. Then, feature extraction is performed based on vehicle image data and a preset world model to obtain the current frame feature map and the target future frame feature map. This step can identify and predict the dynamic changes of the vehicle's surrounding environment and provide key visual information for trajectory planning. Finally, the vehicle's trajectory data can be predicted by combining the vehicle planning data, the current frame feature map and the target future frame feature map, which provides an accurate navigation path for autonomous driving control and ensures that the vehicle can travel safely and efficiently. This application achieves comprehensive perception of the vehicle's surrounding environment and effective prediction of future states by acquiring multi-source data and performing deep fusion and feature extraction, thereby improving the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A flowchart diagram of the first embodiment of the automatic driving control method based on the guidance of the potential world model of the present application is provided;

[0051] Figure 2 A flowchart diagram of the second embodiment of the automatic driving control method based on the guidance of the potential world model of the present application is provided;

[0052] Figure 3 A schematic diagram of a brief process of an automatic driving control method guided by a potential world model according to an embodiment of the present application;

[0053] Figure 4 This is a schematic diagram of the module structure of an automatic driving control device guided by a potential world model according to an embodiment of the present application;

[0054] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the automatic driving control method guided by the potential world model in the embodiment of the present application.

[0055] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] With the development of autonomous driving technology, higher requirements are placed on the decision-making and planning capabilities of vehicles. In complex traffic environments, vehicles need to be able to accurately predict and respond to future changes in the environment to ensure driving safety and efficiency. At present, most end-to-end planning methods in the field of autonomous driving rely on data from the current frame to make decisions and lack the ability to predict future environmental states. These methods usually only consider immediate sensor input and vehicle status, without making full use of historical data and environmental information to predict future changes. This method may perform well in static or slowly changing environments, but in dynamic traffic environments, due to the lack of foresight of upcoming events, it is often unable to make timely and accurate responses.

[0059] The main solution of the embodiment of the present application is: the embodiment first provides comprehensive environmental perception and vehicle status information for the automatic driving system by collecting sensor data, vehicle image data and vehicle navigation information, which is the basis for accurate decision-making and planning. Secondly, the sensor data and vehicle navigation information are used for data fusion to generate vehicle planning data, which helps to integrate the historical and real-time status of the vehicle, as well as the predetermined navigation path, and provides rich context information for subsequent feature extraction and trajectory prediction. Then, feature extraction is performed based on the vehicle image data and the preset world model to obtain the current frame feature map and the target future frame feature map. This step can identify and predict the dynamic changes of the vehicle's surrounding environment and provide key visual information for trajectory planning. Finally, the vehicle's trajectory data can be predicted by combining the vehicle planning data, the current frame feature map and the target future frame feature map, which provides an accurate navigation path for automatic driving control and ensures that the vehicle can travel safely and efficiently. By acquiring multi-source data and performing deep fusion and feature extraction, the embodiment achieves comprehensive perception of the vehicle's surrounding environment and effective prediction of future states, thereby improving the safety of automatic driving.

[0060] It should be noted that the execution subject of the embodiment of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an electronic control unit, etc. The following takes the electronic control unit as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the embodiment of the present application provides an automatic driving control method based on the guidance of the potential world model, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the autonomous driving control method guided by the potential world model of the present application.

[0062] In this embodiment, the automatic driving control method based on the guidance of the potential world model includes steps S10 to S40:

[0063] Step S10, acquiring sensor data, vehicle image data and vehicle navigation information;

[0064] It should be noted that sensor data can be collected by various sensors carried by the vehicle, including but not limited to radar, laser radar (LiDAR), camera, ultrasonic sensor, inertial measurement unit (IMU), etc. Vehicle image data can come from the camera on the vehicle, which can be images captured by a monocular, binocular or multi-camera system. Vehicle navigation information can include GPS, on-board maps, route planning, etc.

[0065] It is understandable that the real-time physical information of the surrounding environment is collected by various sensors carried by the vehicle (such as radar, lidar, camera, etc.), the road visual information captured by the on-board camera, and the path planning and location data provided by the GPS and navigation system are used to provide comprehensive environmental perception and navigation guidance for autonomous driving vehicles, ensuring that the vehicle can drive safely.

[0066] Step S20, performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data;

[0067] It should be noted that vehicle planning data can be a set of comprehensive data obtained by integrating sensor data and vehicle navigation information, which is used to guide the driving decisions and path planning of autonomous vehicles. By integrating these data, a comprehensive and real-time updated vehicle planning data set can be obtained, which is the basis for achieving accurate trajectory planning and safe autonomous driving control. After processing and analysis, these data can help the system make decisions such as changing lanes, accelerating, decelerating or stopping.

[0068] It can be understood that vehicle planning data can be a set of information obtained by comprehensively analyzing and integrating the real-time environmental perception information collected by sensors and the location, route guidance and other data provided by the vehicle navigation system. This set of information includes the vehicle's historical status, current status and expected path, which is used to assist the autonomous driving system in path planning and driving decisions, ensuring that the vehicle can travel safely and efficiently along the predetermined route.

[0069] Step S30, performing feature extraction based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map;

[0070] It should be noted that the preset world model can be a model used to describe the environment in which the autonomous driving vehicle is located (including elements such as roads, traffic signs, other vehicles and pedestrians). This model provides a framework to help understand and interpret the data collected by sensors and cameras. The current frame feature map can be a representation of the features extracted from the vehicle image data, reflecting the visual features of the environment around the vehicle at the current moment. By applying computer vision and deep learning technology, key elements such as lane lines, traffic signs, other vehicles, etc. can be identified and marked, and this information can be encoded into a feature map for current decision-making and planning. The target future frame feature map can be a feature map that predicts the environment around the vehicle at a certain moment in the future based on the current frame feature map and the vehicle's motion state. By simulating the expected movement of the vehicle and changes in the environment, the features of future frames can be predicted, which is crucial for planning the vehicle's action path in advance and making timely responses.

[0071] It can be understood that by utilizing the image information captured by the vehicle's onboard camera and a predefined environmental model, key elements such as lane lines, traffic signs, and other vehicles are identified from the image through computer vision and deep learning technology, and a current frame feature map describing the current environmental state is generated. A target future frame feature map that predicts future environmental changes is also generated. These feature maps provide autonomous driving vehicles with an in-depth understanding of the surrounding environment and predictions of future states, thereby assisting in accurate path planning and driving decisions.

[0072] As an example, feature extraction is performed based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map, including: feature extraction is performed based on the vehicle image data and a preset world model to obtain a historical frame feature map; vehicle trajectory data is recorded through a network training method to obtain a vehicle trajectory feature map, wherein the vehicle trajectory data is obtained based on the sensor; the vehicle trajectory feature map and the historical frame feature map are time-sequentially spliced ​​to obtain a current frame feature map; and the current frame feature map is decoded to obtain a target future frame feature map.

[0073] Among them, the historical frame feature map can be a set of features extracted from the historical image data captured by the vehicle camera, reflecting the environmental state of the vehicle at the past moment. Through the deep learning model, it can identify key elements in the image, such as lane lines, traffic signs, pedestrians, other vehicles, etc., and encode them into feature maps. These feature maps contain the visual information of the vehicle in the historical frame, providing a basis for understanding the environment in which the vehicle is located. The vehicle trajectory feature map can be a trajectory feature representation obtained after processing the vehicle motion trajectory information recorded based on sensor data (such as GPS, IMU, etc.) through a network training method. It not only contains the vehicle's position and speed information, but also contains dynamic information such as acceleration and heading angle. This information is encoded into a feature map to describe the vehicle's motion state. The vehicle trajectory feature map helps the system understand the vehicle's historical motion pattern and predict possible future trajectories.

[0074] Specifically, key visual information is extracted from past images captured by the vehicle's camera; the vehicle trajectory data is recorded through a network training method to obtain a vehicle trajectory feature map, which is derived from the sensor; the vehicle trajectory feature map is spliced ​​with the historical frame feature map in chronological order to form a current frame feature map, which integrates the vehicle's historical position and visual information; finally, the current frame feature map is decoded to predict the target future frame feature map, that is, to foresee the vehicle's future position and environmental changes, and provide decision support for autonomous driving.

[0075] As an example, the current frame feature map is decoded to obtain a target future frame feature map, including: obtaining a preset noise value based on the dimension of the current feature map; performing convolution encoding on the current frame feature map to obtain a reference current frame feature map; performing feature data constraints on the preset noise value to obtain a noise weight value; combining the reference current frame feature map, the noise weight value and the preset noise value through a gated cycle unit to obtain an initial future frame feature map; and adjusting the initial future frame feature map to obtain a target future frame feature map.

[0076] Among them, the preset noise value can be a random noise artificially added during the feature extraction and prediction process, which is used to simulate the uncertainty and changes in the real world and enhance the robustness of the model. The reference current frame feature map can be an image obtained by convolution encoding the current frame feature map, which serves as the basis for future frame prediction and contains key information of the current environment. The noise weight value can be a weight obtained after the feature data constraint is performed on the preset noise value, which is used to adjust the impact of noise on the prediction result and ensure the accuracy of the prediction. The gated recurrent unit (GRU) can be a variant of a recurrent neural network (RNN) for processing time series data, which can capture long-term dependencies and is used to predict future frame features in combination with the current frame and historical information. The initial future frame feature map can be a preliminary predicted future frame feature map obtained by combining the reference current frame feature map, the noise weight value and the preset noise value through GRU. The target future frame feature map can be the final prediction result obtained after further adjustment and optimization of the initial future frame feature map, which more accurately reflects the expected future environmental state.

[0077] Specifically, first, a preset noise value is determined according to the dimension of the current feature map, and then the current frame feature map is convolutionally encoded to generate a reference current frame feature map; then, a feature data constraint is imposed on the preset noise value to calculate a noise weight value, and then a gated recurrent unit (GRU) is used to combine the reference current frame feature map, the noise weight value and the preset noise value to predict an initial future frame feature map, and finally, this initial prediction is adjusted to generate a more accurate target future frame feature map, thereby providing the autonomous driving vehicle with prediction information on the future environment.

[0078] As an example, the initial future frame feature map is adjusted to obtain a target future frame feature map, including: obtaining current frame lidar data based on the sensor data; performing voxel feature encoding on the current frame lidar data to obtain a future frame lidar feature map; obtaining a feature loss value through the future frame lidar feature map and the future frame feature map; and adjusting the initial future frame feature map according to the feature loss value to obtain a target future frame feature map.

[0079] Among them, the current frame lidar data can be the three-dimensional point cloud data of the environment captured by the lidar sensor at the current moment. The lidar determines the distance and shape of the surrounding objects by emitting laser pulses and measuring the time of reflection, generating accurate spatial information. The future frame lidar feature map can be obtained by voxel feature encoding of the current frame lidar data, and is used to predict the state of the environment at a certain moment in the future. Voxel feature encoding is a method of discretizing a continuous three-dimensional space and encoding it into a feature map, which is convenient for computer processing and analysis. The future frame lidar feature map reflects the system's prediction of the position and shape of objects in the next environment, and is an important basis for the decision-making and planning of the autonomous driving system. The feature loss value can be an indicator used in machine learning to measure the difference between the predicted feature map and the actual feature map. By comparing the future frame lidar feature map (predicted feature map) and the actually observed future frame feature map (real feature map), a loss value is calculated to indicate the accuracy of the prediction.

[0080] Specifically, the sensor data is first used to obtain the lidar data of the current frame, and then the voxel features of this data are encoded to generate a future frame lidar feature map, which predicts the environment layout at the future moment; then, the feature loss value is calculated by comparing the future frame lidar feature map with the future frame feature map generated by other sensor data. This loss value reflects the difference between the two; finally, the initially predicted future frame feature map is adjusted according to the feature loss value to optimize and correct the prediction result, thereby obtaining a more accurate target future frame feature map to provide support for the decision-making of the autonomous driving vehicle.

[0081] Step S40, obtaining predicted trajectory data based on the vehicle planning data, the current frame feature map and the target future frame feature map, so that the vehicle completes automatic driving control according to the predicted trajectory data.

[0082] It should be noted that the predicted trajectory data refers to a series of predicted information about the future position and motion state of the vehicle obtained through comprehensive analysis of the vehicle planning data, the current frame feature map, and the target future frame feature map in the autonomous driving system. These data include but are not limited to: expected path, the path or trajectory that the vehicle should follow in the future; speed and acceleration, the expected speed and acceleration of the vehicle at different time points; steering angle, the steering wheel angle that the vehicle needs to adjust during driving; obstacle avoidance strategy, the strategy adopted to avoid collisions with other vehicles, pedestrians or obstacles; traffic rule compliance, the action plan to ensure that the vehicle complies with traffic signals, signs and lane rules.

[0083] It can be understood that the comprehensive vehicle planning data, current frame feature map and target future frame feature map are used to analyze the vehicle's current state, surrounding environment and expected future changes to generate predictive information about the vehicle's future position, speed, path, etc. These data guide the autonomous driving system to achieve automatic navigation and control of the vehicle, ensuring that the vehicle can safely travel along the predetermined trajectory.

[0084] As an example, based on the vehicle planning data, the current frame feature map and the target future frame feature map, the predicted trajectory data is obtained, including: obtaining a preset transformer model; introducing a cross-attention mechanism based on the preset transformer model, and obtaining the predicted trajectory data through the vehicle planning data, the current frame feature map and the target future frame feature map.

[0085] The preset transformer model may be a predefined and trained transformer model used in an autonomous driving system to generate predicted trajectory data. The transformer model is a deep learning architecture originally designed to process sequence data, especially in the field of natural language processing to understand the structure and meaning of language. In the field of autonomous driving, the transformer model is used to process multimodal data, such as vehicle planning data, current frame feature map, and target future frame feature map, to achieve prediction of the vehicle's future trajectory. The preset transformer model can process multimodal data by introducing a cross-attention mechanism, and fuse data from different sources and types (such as sensor data, image data, etc.) together to obtain a more comprehensive understanding and prediction. It can also capture time series information. Through the attention mechanism, the model can capture key information in the time series and understand the relationship between data at different time points. The prediction accuracy can be improved. By learning the complex relationship between vehicle planning data, current frame feature map, and target future frame feature map, the model can more accurately predict the vehicle's future trajectory. The specific predicted trajectory data may be:

[0086]

[0087] Where T final is the predicted trajectory data, is the vehicle planning data, It is a refined BEV feature.

[0088] Specifically, by introducing the cross-attention mechanism through the preset transformer model, the vehicle planning data, the current frame feature map and the target future frame feature map can be comprehensively analyzed to generate predicted trajectory data. These data guide the autonomous driving system to predict the future position and motion trajectory of the vehicle to achieve safe autonomous driving control.

[0089] This embodiment provides an automatic driving control method based on the guidance of a potential world model. This embodiment first provides comprehensive environmental perception and vehicle status information for the automatic driving system by collecting sensor data, vehicle image data and vehicle navigation information, which is the basis for accurate decision-making and planning. Secondly, the sensor data and vehicle navigation information are used for data fusion to generate vehicle planning data, which helps to integrate the historical and real-time status of the vehicle and the predetermined navigation path, and provides rich context information for subsequent feature extraction and trajectory prediction. Then, feature extraction is performed based on the vehicle image data and the preset world model to obtain the current frame feature map and the target future frame feature map. This step can identify and predict the dynamic changes of the vehicle's surrounding environment and provide key visual information for trajectory planning. Finally, the vehicle planning data, the current frame feature map and the target future frame feature map are combined to predict the vehicle's trajectory data, which provides an accurate navigation path for automatic driving control and ensures that the vehicle can travel safely and efficiently. This embodiment achieves comprehensive perception of the vehicle's surrounding environment and effective prediction of future states by acquiring multi-source data and performing deep fusion and feature extraction, thereby improving the safety of automatic driving.

[0090] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , Figure 2 This is a flow chart of a second embodiment of an autonomous driving control method based on guidance of a potential world model of the present application. Step S20 of the autonomous driving control method based on guidance of a potential world model includes steps S21 to S22:

[0091] Step S21, obtaining position data, speed data, heading angle data and vehicle size data based on the sensor data;

[0092] It should be noted that the location data can be the specific coordinates of the vehicle on the road, usually provided by GPS or other positioning systems, including longitude and latitude information. The speed data can be the current speed of the vehicle, which can be measured directly by a speed sensor or calculated through changes in the location data. The heading angle data can be the direction of travel of the vehicle, that is, the angle of the vehicle relative to the north or the road, usually provided by an inertial measurement unit (IMU). The vehicle size data can be the physical size of the vehicle, including length, width and height, which is essential for avoiding collisions, passing through narrow spaces and complying with traffic rules.

[0093] It is understandable that the vehicle's real-time geographic location, driving speed, driving direction and three-dimensional size information collected by on-board sensors such as GPS, speedometer, gyroscope and lidar provide the vehicle's specific state and size in the physical world, which is the basis for accurate path planning and safe driving decisions.

[0094] Step S22, stacking the position data, the speed data, the heading angle data and the vehicle size data, and encoding the data through a multi-layer perceptron structure to obtain vehicle history data;

[0095] It should be noted that the multilayer perceptron structure (Multilayer Perceptron, MLP) can be an artificial neural network composed of multiple layers, including an input layer, one or more hidden layers and an output layer. Each layer is composed of multiple neurons, which are connected by weights and process information through activation functions. MLP can learn the nonlinear mapping from input data to output data. It is a commonly used feedforward neural network suitable for various classification and regression tasks. Vehicle history data can be a set of data obtained by integrating and encoding vehicle status information such as position data, speed data, heading angle data and vehicle size data. Stacking can be integrating vehicle status data from different sources to form a unified data structure for easy processing and analysis. Data encoding can be using a multilayer perceptron structure to extract and transform features of stacked data, converting raw data into higher-level and more abstract feature representations, which can better reflect the historical behavior and state changes of the vehicle. Vehicle history data can be encoded data representing the historical information of the vehicle, including its past motion trajectory, speed changes, heading changes, etc., which are essential for understanding the behavior pattern of the vehicle, predicting future states, and performing effective path planning. The calculation formula for specific vehicle history data can be:

[0096]

[0097] in is the vehicle history data V x is the current horizontal coordinate position, V y is the current ordinate position, Δx is the acceleration in the abscissa direction, Δy is the acceleration in the ordinate direction, V is the current vehicle speed, H is the vehicle length, W is the vehicle width, and S is the heading angle.

[0098] It is understandable that these different vehicle status information are first integrated together to form a comprehensive data set; then the stacked data are feature extracted and encoded through a multi-layer perceptron structure and converted into higher-dimensional and more abstract vehicle history data. These data can more comprehensively reflect the vehicle's past motion characteristics and behavior patterns, and provide support for the decision-making of the autonomous driving system.

[0099] Step S23, performing data fusion based on the vehicle historical data and the vehicle navigation information to obtain vehicle planning data.

[0100] It can be understood that the vehicle's past driving data, including historical information such as position, speed, and heading angle, is combined with the future path planning information provided by its navigation system, and through certain data processing and analysis techniques, a comprehensive data set is generated. This data set is called vehicle planning data, which includes both the vehicle's historical behavior patterns and the predetermined driving routes and goals, providing decision support for autonomous driving and helping the vehicle achieve more accurate path planning and driving control.

[0101] As an example, data fusion is performed based on the vehicle historical data and the vehicle navigation information to obtain vehicle planning data, including: data encoding the vehicle navigation data through a multi-layer perceptron structure to obtain vehicle navigation data; calculating the vehicle navigation data and the vehicle historical data through a neural network algorithm to obtain a weight vector and a scaling vector; and obtaining vehicle planning data based on the weight vector, the scaling vector, the vehicle historical data and the vehicle navigation data.

[0102] Among them, the weight vector can be a series of parameters that connect the input and output in the neural network, and each weight represents the degree of influence of the input feature on the output result. In a multi-layer perceptron (MLP) or other neural network structure, the weight determines how the input signal is weighted and combined to produce the output. In the encoding process of vehicle navigation data, the weight vector is obtained through a learning process, which can convert navigation data (such as destination location, path information, etc.) into a feature representation that is instructive for vehicle planning. The scaling vector is usually used to adjust or normalize the output in the neural network to ensure that the data is within a suitable range to improve the stability and performance of the model. In the process of generating vehicle planning data, the scaling vector may be used to adjust the influence of the weight vector, or to normalize the fusion results of vehicle history data and navigation data to ensure that data from different sources have consistent proportions and scales in the feature space. The calculation formula for vehicle navigation data and vehicle planning data can be:

[0103]

[0104] in It is the vehicle navigation data. Linear represents the fully connected layer in pytorch calculation. Here we use Linear to generate the weight of the mixed trajectory Query (the first Linear calculation) and the scaling (the second Linear calculation). Planning data for the vehicle.

[0105] Specifically, the vehicle navigation data is first encoded using a multi-layer perceptron structure and converted into a format suitable for neural network processing; the vehicle navigation data is then combined with the vehicle history data through a neural network algorithm to calculate the weight vector and scaling vector, which are used to adjust the importance of features and normalize the data scale, respectively; finally, the weight vector, scaling vector, vehicle history data and vehicle navigation data are combined to generate comprehensive vehicle planning data, which provides the autonomous driving vehicle with key information on how to make driving decisions based on historical behavior and predetermined navigation paths.

[0106] This embodiment first provides the automatic driving system with accurate vehicle status and physical characteristics information through the position, speed, heading angle and vehicle size data collected by sensors, which is the basis for subsequent analysis and decision-making. Then these data are stacked and encoded through a multi-layer perceptron structure. The generated vehicle history data can reflect the vehicle's past motion trajectory and behavior pattern in detail, providing rich features for understanding the dynamic characteristics of the vehicle. Finally, the vehicle history data is fused with the navigation information, and the obtained planning data not only includes the vehicle's historical behavior, but also integrates the predetermined driving route and target, providing comprehensive decision support information for the automatic driving system. This embodiment can generate a comprehensive vehicle planning data set based on real-time sensor data and historical behavior data. This data set can help more accurately predict the future state of the vehicle, more effectively plan the driving path, and make more reasonable driving decisions. In this way, the automatic driving system can improve the safety, efficiency and comfort of driving, while reducing the errors and uncertainties of human operation, and achieve a more intelligent and automated driving experience.

[0107] For example, in order to help understand the implementation process of the automatic driving control method based on the guidance of the potential world model obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flowchart of an autonomous driving control method based on the guidance of a potential world model is provided. Specifically:

[0108] First, data is collected through sensors, vehicle images, and navigation information. Then, the position, speed, heading angle, and vehicle size data are encoded using a multi-layer perceptron structure to generate vehicle history data, which is then fused with the navigation information encoded after feature extraction by the motion layer neural network to obtain vehicle planning data. At the same time, feature maps are extracted from vehicle image data, and vehicle trajectory data is recorded through a network training method. These data are spliced ​​in time series to obtain the current frame feature map, which is then decoded through a potential world model to obtain the initial future frame feature map. Next, the lidar data is voxel-featured to extract lidar features to obtain the loss value, which is adjusted based on the loss value and the initial future frame feature map to obtain the future frame feature map. Finally, the cross-attention mechanism in the transformer model is used to integrate the vehicle planning data, the current frame feature map, and the target future frame feature map to generate predicted trajectory data and realize autonomous driving control.

[0109] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the autonomous driving control method of the present application guided by the potential world model. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0110] This application also provides an automatic driving control device based on the guidance of the potential world model, please refer to Figure 4 , the automatic driving control device based on the guidance of the potential world model includes:

[0111] An acquisition module 10, for acquiring sensor data, vehicle image data and vehicle navigation information;

[0112] An obtaining module 20 is used to perform data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data;

[0113] A fusion module 30 is used to extract features based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map;

[0114] The completion module 40 is used to obtain predicted trajectory data based on the vehicle planning data, the current frame feature map and the future frame feature map, so that the vehicle can complete the automatic driving control according to the predicted trajectory data.

[0115] The automatic driving control device based on the guidance of the potential world model provided by the present application adopts the automatic driving control method based on the guidance of the potential world model in the above-mentioned embodiment, which can solve the technical problem of how to improve the automatic driving system's ability to predict future environmental changes. Compared with the prior art, the beneficial effects of the automatic driving control device based on the guidance of the potential world model provided by the present application are the same as the beneficial effects of the automatic driving control method based on the guidance of the potential world model provided by the above-mentioned embodiment, and the other technical features of the automatic driving control device based on the guidance of the potential world model are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0116] The present application provides an autonomous driving control device based on the guidance of a potential world model, and the autonomous driving control device based on the guidance of a potential world model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the autonomous driving control method based on the guidance of the potential world model in the above-mentioned embodiment one.

[0117] Reference below Figure 5 , which shows a schematic diagram of the structure of an automatic driving control device based on potential world model guidance suitable for implementing the embodiment of the present application. The automatic driving control device based on potential world model guidance in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The autonomous driving control device based on potential world model guidance shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0118] like Figure 5As shown, the automatic driving control device guided by the potential world model may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the automatic driving control device guided by the potential world model are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the autonomous driving control device guided by the potential world model to communicate wirelessly or by wire with other devices to exchange data. Although the figure shows an autonomous driving control device guided by the potential world model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0119] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0120] The autonomous driving control device based on the guidance of the potential world model provided by the present application adopts the autonomous driving control method based on the guidance of the potential world model in the above-mentioned embodiment, which can solve the technical problem of how to improve the autonomous driving system's ability to predict future environmental changes. Compared with the prior art, the beneficial effects of the autonomous driving control device based on the guidance of the potential world model provided by the present application are the same as the beneficial effects of the autonomous driving control method based on the guidance of the potential world model provided by the above-mentioned embodiment, and the other technical features of the autonomous driving control device based on the guidance of the potential world model are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0121] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0123] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the autonomous driving control method based on potential world model guidance in the above-mentioned embodiment.

[0124] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0125] The above-mentioned computer-readable storage medium may be included in the autonomous driving control device guided by the potential world model; or it may exist independently without being assembled into the autonomous driving control device guided by the potential world model.

[0126] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the autonomous driving control device guided by the potential world model, the autonomous driving control device guided by the potential world model: obtains sensor data, vehicle image data and vehicle navigation information; performs data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data; performs feature extraction based on the vehicle image data and the preset world model to obtain a current frame feature map and a target future frame feature map; obtains predicted trajectory data based on the vehicle planning data, the current frame feature map and the target future frame feature map, so that the vehicle completes autonomous driving control according to the predicted trajectory data.

[0127] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0130] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned automatic driving control method guided by the potential world model, and can solve the technical problem of how to improve the automatic driving system's ability to predict future environmental changes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the automatic driving control method guided by the potential world model provided by the above-mentioned embodiment, and will not be repeated here.

[0131] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned automatic driving control method based on guidance of a potential world model.

[0132] The computer program product provided by this application can solve the technical problem of how to improve the ability of the autonomous driving system to predict future environmental changes. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the autonomous driving control method based on the guidance of the potential world model provided in the above embodiment, and will not be repeated here.

[0133] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An automatic driving control method based on guidance of a potential world model, characterized in that: The method comprises: Acquire sensor data, vehicle image data, and vehicle navigation information; Performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data; Perform feature extraction based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map; Based on the vehicle planning data, the current frame feature map and the target future frame feature map, predicted trajectory data is obtained so that the vehicle can complete automatic driving control according to the predicted trajectory data.

2. The method according to claim 1, characterized in that The step of fusing the sensor data and the vehicle navigation information to obtain vehicle planning data includes: Obtaining position data, speed data, heading angle data and vehicle size data based on the sensor data; The position data, the speed data, the heading angle data and the vehicle size data are stacked, and data encoding is performed through a multi-layer perceptron structure to obtain vehicle history data; Data fusion is performed based on the vehicle historical data and the vehicle navigation information to obtain vehicle planning data.

3. The method according to claim 2, characterized in that The step of fusing the vehicle historical data and the vehicle navigation information to obtain vehicle planning data includes: The vehicle navigation data is encoded by a multi-layer perceptron structure to obtain the vehicle navigation data; Calculate the vehicle navigation data and the vehicle history data by a neural network algorithm to obtain a weight vector and a scaling vector; Vehicle planning data is obtained based on the weight vector, the scaling vector, the vehicle history data, and the vehicle navigation data.

4. The method according to claim 1, characterized in that The feature extraction based on the vehicle image data and the preset world model to obtain the current frame feature map and the target future frame feature map includes: Perform feature extraction based on the vehicle image data and a preset world model to obtain a historical frame feature map; Recording vehicle trajectory data through a network training method to obtain a vehicle trajectory feature graph, wherein the vehicle trajectory data is obtained based on the sensor; The vehicle trajectory feature map and the historical frame feature map are spliced ​​in time sequence to obtain a current frame feature map; The current frame feature map is decoded to obtain a target future frame feature map.

5. The method according to claim 4, characterized in that The decoding of the current frame feature map to obtain a target future frame feature map includes: Based on the dimension of the current feature map, obtaining a preset noise value; Performing convolution encoding on the current frame feature map to obtain a reference current frame feature map; Performing feature data constraints on the preset noise value to obtain a noise weight value; Obtaining an initial future frame feature map by combining the reference current frame feature map, the noise weight value and the preset noise value through a gated cycle unit; The initial future frame feature map is adjusted to obtain a target future frame feature map.

6. The method according to claim 5, characterized in that The adjusting the initial future frame feature map to obtain a target future frame feature map includes: Obtaining current frame laser radar data based on the sensor data; Performing voxel feature encoding on the current frame laser radar data to obtain a future frame laser radar feature map; Obtaining a feature loss value through the future frame laser radar feature map and the future frame feature map; The initial future frame feature map is adjusted according to the feature loss value to obtain a target future frame feature map.

7. The method according to claim 1, characterized in that The step of obtaining predicted trajectory data based on the vehicle planning data, the current frame feature map, and the target future frame feature map includes: Get the preset transformer model; A cross attention mechanism is introduced based on the preset transformer model, and predicted trajectory data is obtained through the vehicle planning data, the current frame feature map and the target future frame feature map.

8. An automatic driving control device based on guidance of a potential world model, characterized in that: The device comprises: An acquisition module, used to acquire sensor data, vehicle image data and vehicle navigation information; An obtaining module, used for performing data fusion based on the sensor data and the vehicle navigation information to obtain vehicle planning data; A fusion module, used for extracting features based on the vehicle image data and a preset world model to obtain a current frame feature map and a target future frame feature map; The completion module is used to obtain predicted trajectory data based on the vehicle planning data, the current frame feature map and the future frame feature map, so that the vehicle can complete the automatic driving control according to the predicted trajectory data.

9. An automatic driving control device guided by a potential world model, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the automatic driving control method based on potential world model guidance as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the automatic driving control method based on potential world model guidance as described in any one of claims 1 to 7 are implemented.

Images