Driving control method, electronic equipment and storage medium

Abstract

The invention provides a driving control method, electronic equipment and a storage medium, and the driving control method comprises the steps: inputting image data collected by a vehicle into a generative model, and obtaining entity information around the vehicle; inputting the entity information, the vehicle data of the vehicle and session data obtained in advance into the generative model to obtain a driving behavior of the vehicle; inputting the image data and the vehicle data into a planning model to obtain an initial trajectory of the vehicle; using a preset optimizer to optimize the initial trajectory according to the driving behavior and preset traffic rule information to obtain an optimized target trajectory and control parameters; and controlling the vehicle to move along the target track based on the control parameters. The accuracy of controlling the vehicle to run along the target track can be improved.

Description

Technical Field

[0001] This application belongs to the field of vehicles and relates to autonomous driving technology, and more particularly to a driving control method, electronic device and storage medium. Background Technology

[0002] With the increasing number of motor vehicles, the driving environment is becoming increasingly complex. As the transportation industry advances, the autonomous driving functions within vehicle systems are constantly improving. Currently, with the emergence of large language models and visual foundational models, multimodal AI systems benefiting from these models possess the ability to perceive the real world and make decisions. Large language models have received widespread attention in autonomous driving planning and interpretability. However, implementing autonomous driving control based on large language models presents challenges such as long response times and difficulties in quantifying decision-making information. Summary of the Invention

[0003] In view of the above, it is necessary to propose a driving control method, electronic device and storage medium to solve the technical problem that the decision response time of autonomous driving of vehicles is long and difficult to quantify.

[0004] This application provides a driving control method, the method comprising: inputting image data collected by a vehicle into a generative model to obtain entity information around the vehicle; inputting the entity information, vehicle data, and pre-acquired session data into the generative model to obtain the driving behavior of the vehicle; inputting the image data and the vehicle data into a planning model to obtain the initial trajectory of the vehicle; using a preset optimizer, optimizing the initial trajectory according to the driving behavior and preset traffic rule information to obtain an optimized target trajectory and control parameters; and controlling the vehicle to move along the target trajectory based on the control parameters.

[0005] In some embodiments, the method further includes training the optimizer, comprising: determining the historical environmental features of the vehicle using the generative model based on the vehicle's historical image data; inputting the historical environmental features, the vehicle's historical vehicle data, and historical session data into the generative model to obtain the vehicle's behavior data; inputting the historical image data and the historical vehicle data into the planning model to obtain the vehicle's trajectory data; updating the trajectory data using a pre-built initial optimizer based on the behavior data and the traffic rule information to obtain an updated predicted trajectory and prediction parameters; determining the loss value of the initial optimizer based on the behavior data, the trajectory data, the historical environmental features, the predicted trajectory, and the prediction parameters; updating the initial optimizer based on the loss value; stopping the updating of the initial optimizer when the loss value meets a preset first termination condition, thereby obtaining an optimizer trained to a convergent state.

[0006] In some embodiments, determining the loss value of the initial optimizer based on the behavioral data, the trajectory data, the historical environmental features, the predicted trajectory, and the prediction parameters includes: determining a first loss of the initial optimizer based on the trajectory data, the prediction parameters, the predicted trajectory, the historical environmental features, and the traffic rule information; determining a second loss of the initial optimizer based on the behavioral data, the prediction parameters, the predicted trajectory, and the historical environmental features; and performing a weighted summation operation on the first loss and the second loss to obtain the loss value of the initial optimizer.

[0007] In some embodiments, determining the first loss of the initial optimizer based on the trajectory data, the prediction parameters, and the traffic rule information includes: determining a first speed loss based on the difference between the predicted speed in the prediction parameters and the speed limit in the traffic rule information; determining a first position of any entity around the vehicle in the coordinate system corresponding to the vehicle based on the historical environmental features; and determining a second position of the vehicle in the coordinate system based on the predicted trajectory; determining a position loss based on the difference between the first position and the second position; determining a dynamic loss based on the acceleration in the prediction parameters, the rate of change of the acceleration, the steering angle in the prediction parameters, and the rate of change of the steering angle; determining a lane loss based on the lane position information indicated by the historical environmental features and the difference between the second position; and determining the first loss of the initial optimizer based on the speed loss, the position loss, the dynamic loss, and the lane loss.

[0008] In some embodiments, determining the second loss of the initial optimizer based on the behavioral data, the prediction parameters, the predicted trajectory, and the historical environmental features includes: determining a second speed loss based on the speed change data of the vehicle indicated by the behavioral data, and the difference between the predicted speed and the speed of any entity around the vehicle indicated by the historical environmental features; determining a steering loss based on the steering state of the vehicle indicated by the behavioral data, and the difference between the first position and the second position; and determining the second loss of the initial optimizer based on the second speed loss and the steering loss.

[0009] In some embodiments, determining the driving behavior of the vehicle using the generative model based on the environmental features, the vehicle data, and pre-acquired session data includes: formatting the environmental features, the vehicle data, and the session data according to the logical chain information in preset prompt words to obtain prompt information; and determining the corresponding driving behavior based on the prompt information using the generative model.

[0010] In some embodiments, inputting the image data and the vehicle data into a planning model to obtain the initial trajectory of the vehicle includes: using a first encoder in the planning model to extract features from the state information in the vehicle data to obtain vectorized motion features of the vehicle; using a second encoder in the planning model to extract features from the image data to obtain top-down features of the vehicle; using a third encoder in the planning model to extract features from the map information in the vehicle data to obtain map features of the area where the vehicle is located; and using a fourth encoder in the planning model to predict the initial trajectory of the vehicle based on the motion features, the top-down features, and the map features.

[0011] In some embodiments, updating the initial trajectory using a preset optimizer based on the driving behavior and pre-stored traffic rule information to obtain an updated target trajectory and control parameters includes: predicting the predicted position of the vehicle using the optimizer based on the driving behavior and pre-stored traffic rule information; updating the initial trajectory based on the deviation between the initial trajectory and the predicted position until the deviation meets a preset second termination condition; and obtaining the target trajectory and control parameters based on the updated initial trajectory.

[0012] This application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instructions stored in the memory to implement the driving control method.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the driving control method.

[0014] As can be seen from the above technical solutions, the embodiments of this application extract environmental features based on image data and generative models, which can ensure the integrity of environmental information and provide data support for driving trajectory decision-making. These environmental features, including distance, dynamics, and scene association, can directly provide understandable and usable input for driving behavior decision-making, avoiding information gaps between perceived information and decision-making needs. Furthermore, the generative model has context modeling capabilities, which can cope with extreme scenarios where traditional perception is prone to failure, improving the credibility and stability of environmental features. In addition, determining driving behavior based on environmental features, vehicle data, and session data ensures that the generated driving behavior conforms to the collaborative adaptation between the user, vehicle, and environment, preventing the subsequently generated target trajectory and control parameters from deviating from actual user decisions. Moreover, determining driving behavior based on environmental information constraints avoids a disconnect between decision-making and vehicle physical capabilities, reducing the probability of dangerous operations. Predicting the initial trajectory based on image data and vehicle data, and optimizing the initial trajectory based on driving behavior and traffic rules, ensures the real-time performance and reliability of trajectory planning through a coarse-filtering planning and fine-tuning optimization mechanism. This ensures that trajectory planning is synchronized with real-time road condition changes, avoiding the risk of accidents caused by missed lane-changing opportunities due to computational delays. It can ensure the compliance and safety of the final optimized target trajectory, and improve the accuracy and traceability of controlling the vehicle to move along the target trajectory based on control parameters. Attached Figure Description

[0015] Figure 1 This is an application scenario diagram of a driving control method provided in an embodiment of this application.

[0016] Figure 2 This is a flowchart of a driving control method provided in an embodiment of this application.

[0017] Figure 3 This is a flowchart of a method for determining the driving behavior of a vehicle according to an embodiment of this application.

[0018] Figure 4 This is a flowchart of a method for predicting the initial trajectory of a vehicle according to an embodiment of this application.

[0019] Figure 5 This is a flowchart of a method for training an optimizer provided in an embodiment of this application.

[0020] Figure 6 This is a flowchart of a method for determining the loss value of an optimizer according to an embodiment of this application.

[0021] Figure 7 This is a flowchart of a method for determining the first loss value of an optimizer according to an embodiment of this application.

[0022] Figure 8This is a flowchart of a method for determining a second loss value of an optimizer according to an embodiment of this application.

[0023] Figure 9 This is a flowchart of a method for determining an updated target trajectory and control parameters according to an embodiment of this application.

[0024] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0028] This application provides a driving control method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0030] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0032] like Figure 1 The diagram illustrates an application scenario of a driving control method according to an embodiment of this application. This driving control method can be applied to an electronic device 100. The electronic device 100 is disposed in a vehicle 200 and is communicatively connected to vehicle components 300 within the vehicle 200. The vehicle components 300 may include a camera device, a steering device, and an acceleration device for the vehicle 200. This application does not limit the specific type of the vehicle components 300.

[0033] In this application, electronic device 100 acquires image data collected by vehicle component 300 (e.g., vehicle-mounted camera device) and inputs the image data into a preset generative model to obtain environmental features of the area where vehicle 200 is located. These environmental features are used to indicate entity information of the area where vehicle 200 is located. Entities in the area where vehicle 200 is located may include other vehicles, pedestrians, obstacles, traffic signs, etc., and this application does not limit this.

[0034] In this application, the electronic device 100 further determines the driving behavior of the vehicle 200 using a generative model based on environmental features, vehicle data of the vehicle 200, and pre-acquired session data. The driving behavior refers to actions that control the driving state of the vehicle 200. For example, the driving behavior may include turning left, turning right, overtaking, increasing acceleration, etc.

[0035] In this application, the electronic device 100 also inputs image data and vehicle data of the vehicle 200 into a preset planning model to obtain the initial trajectory output by the planning model. The vehicle data may include the vehicle 200's status information, environmental information, navigation information, and interaction information between the user and the vehicle 200. The environmental information may include map information and traffic reference lines for the area where the vehicle 200 is located. The vehicle 200's status information may include its location (e.g., latitude and longitude), its orientation, and its real-time speed. The vehicle 200's navigation information may include a target reference line for its direction of travel. The interaction information between the user and the vehicle 200 may include voice or text commands issued by the user to the electronic device 100 of the vehicle 200, and may also include information fed back to the user by a large language model running in the electronic device 100 of the vehicle 200 based on the user's commands.

[0036] In this application, the electronic device 100 optimizes the initial trajectory using a preset optimizer based on driving behavior and preset traffic rule information to obtain an optimized target trajectory and control parameters. The vehicle 200 is then controlled to move along the target trajectory based on the control parameters. Since the optimized target trajectory is generated based on constraints from traffic rule information and driving behavior output by a generative model, the target trajectory closely matches the current driving state of the vehicle 200, the environment of the area where the vehicle 200 is located, and traffic rules. Therefore, the accuracy of controlling the vehicle 200 to move along the target trajectory based on the control parameters is high.

[0037] like Figure 2 The diagram shown is a flowchart of a driving control method according to an embodiment of this application. The order of the steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The driving control method provided in this embodiment includes the following steps.

[0038] S20, input the image data collected by the vehicle into the generative model to obtain entity information around the vehicle.

[0039] In one embodiment of this application, in order to plan the vehicle's driving trajectory and improve the accuracy of controlling the vehicle to drive along the planned trajectory, the vehicle's trajectory and control parameters can be predicted by combining data from multiple dimensions of the vehicle, and the vehicle can be controlled to drive along the trajectory according to the control parameters.

[0040] The vehicle's multi-dimensional data can include vehicle image data, vehicle status information, vehicle environmental information, vehicle navigation information, and user-vehicle interaction information.

[0041] The image data can be image data of the vehicle's surroundings captured by the vehicle's onboard camera. Specifically, the image data of the vehicle's surroundings captured by the onboard camera can include image data of the front of the vehicle, the rear of the vehicle, the front left of the vehicle, the rear left of the vehicle, the front right of the vehicle, and the rear right of the vehicle.

[0042] The environmental information may include map information and traffic reference lines for the area where the vehicle is located.

[0043] The vehicle's status information may include its location (e.g., the latitude and longitude of the vehicle's location), the direction the vehicle is facing, and the vehicle's real-time speed.

[0044] The vehicle's navigation information may include target reference lines.

[0045] The interaction information between the user and the vehicle may include voice or text commands issued by the user to the vehicle's electronic devices, as well as information fed back to the user by a large language model running in the vehicle's electronic devices based on the user's commands.

[0046] In one embodiment of this application, image data of the vehicle's surroundings captured by an in-vehicle camera can be analyzed to determine the vehicle's environmental characteristics, and the vehicle's driving trajectory and control parameters can be predicted based on these environmental characteristics, thereby improving the accuracy of the generated vehicle driving trajectory and control parameters.

[0047] Specifically, the environmental features of the vehicle can be determined using a pre-defined generative model based on the vehicle's image data. These environmental features indicate information about entities surrounding the vehicle.

[0048] For example, the environmental features of a vehicle output by a generative model may include: <Type: Vehicle, Status: Driving, Visual Description: Red Sedan, Location: [12,13,11,12], Image: Front, Importance: High>; <Type: Traffic light, Status: null, Visual description: Red light for left turn, green light for straight line, Location: [12,13,11,12], Image: Front, Importance: High>; <Type: Traffic sign, Status: null, Visual description: Speed ​​limit 80km / h, Location: [12,13,11,12], Image: Front left, Importance: High>.

[0049] For example, the environmental features of the vehicle output by the generative model can also include the weather conditions, lane conditions and emergency situations in the area where the vehicle is located. For example, the environmental features can specifically include: <sunny daytime, high visibility>; <water accumulation ahead>; <intersection, construction zone, severe congestion>; <three lanes in total, the left lane is passable, the right lane is not passable>.

[0050] The preset generative model can be any generative model with multimodal information processing capabilities, and this application does not limit the specific type of generative model. For example, the generative model can be the Qwen model, the GPT model, the Gemini model, etc.

[0051] In one embodiment of this application, in order to improve the performance of the generative model and thus improve the accuracy of generating the vehicle's driving trajectory and control parameters using the generative model, the generative model can also be trained based on a pre-stored set of driving data.

[0052] The pre-stored driving data set may include question-and-answer information based on the vehicle's environment during autonomous driving. For example, the driving data set may be the DriveLM dataset.

[0053] Specifically, the generative model can be trained using the DriveLM dataset. The parameters of the generative model can be adjusted using low-rank adaptation (Lora) to ensure that the model can learn representations of entities surrounding the vehicle based on image data. This ensures that the generative model can identify the entities of interest in the vehicle's environment based on image data captured by the onboard camera (e.g., pedestrians, motor vehicles, non-motorized vehicles, traffic lights, traffic signs, etc.).

[0054] S21, input the entity information, the vehicle data, and the pre-acquired session data into the generative model to obtain the vehicle's driving behavior.

[0055] In one embodiment of this application, in order to predict the driving behavior of a vehicle in the future period based on the environmental information of the area where the vehicle is located, the driving behavior of the vehicle can be determined by using a generative model based on environmental characteristics, vehicle data and pre-acquired session data.

[0056] Among them, driving behavior refers to the actions that control the driving state of the vehicle. For example, driving behavior may include turning left, turning right, overtaking, increasing acceleration, etc.

[0057] In one embodiment of this application, environmental features, vehicle data, and session data can be formatted based on prompt words containing logical chain information to obtain prompt information. This ensures that the generative model can understand the information in the environmental features, vehicle data, and session data based on the prompt information, thereby improving the accuracy of the predicted vehicle driving behavior. The driving behavior information can be a collection of multiple vehicle behaviors described in natural language, and the various pieces of information in the driving behavior have a temporal correlation.

[0058] For example, the prompt message may include: Background information: You are driving on a 4-lane road; You are currently driving in the leftmost lane; Your current location is (0.00, 0.00), your speed is 24.03 meters per second, your acceleration is -0.05 meters per second squared, and there are currently 3 lanes available. You can turn left or right.

[0059] There are other vehicles around your vehicle. Here is some basic information about them: Vehicle 1 is traveling in the lane to your right and is in front of you; Vehicle 1 is currently traveling at (278, 96), and this position is to the right front.

[0060] Vehicle 2 is traveling in the same lane as you and is in front of you; Vehicle 2 stops at (222, 0.00), and this position is behind.

[0061] Your optional actions include: Maintain current speed and stay in the current lane. Behavior indicator: 1; Turn right, changing to the right lane of the current lane, behavior sign: 2; Accelerate, control vehicle acceleration, behavior indicator: 3; Decelerate, control the vehicle to slow down, behavior indicator: 4.

[0062] Please output the corresponding driving behavior based on the following content {}.

[0063] For example, the results output by the generative model may include "acceleration, left turn, no overtaking, shortening following distance, and increasing acceleration".

[0064] In one embodiment of this application, the specific method for determining the vehicle's driving behavior using the generative model based on the environmental characteristics, the vehicle's vehicle data, and pre-acquired session data can be found in [link to relevant documentation]. Figure 3 The corresponding detailed explanation.

[0065] S22, input the image data and the vehicle data into the planning model to obtain the initial trajectory of the vehicle.

[0066] In one embodiment of this application, in order to initially determine the initial trajectory of a vehicle, a preset planning model can be used to analyze the features related to vehicle movement in image data and vehicle data, and output the initial trajectory of the vehicle.

[0067] The image data can be image data of the vehicle's surroundings captured by the vehicle's onboard camera. Specifically, the image data of the vehicle's surroundings captured by the onboard camera can include image data of the front of the vehicle, the rear of the vehicle, the front left of the vehicle, the rear left of the vehicle, the front right of the vehicle, and the rear right of the vehicle.

[0068] Vehicle data can include vehicle status information, environmental information, and navigation information.

[0069] Specifically, environmental information can include map information and traffic reference lines for the area where the vehicle is located. Vehicle status information can include the vehicle's location (e.g., latitude and longitude), the vehicle's orientation, and its real-time speed. Vehicle navigation information can include target reference lines.

[0070] In one embodiment of this application, the planning model can be a vehicle data processing framework, including multiple encoders and decoders, for outputting vectorized trajectory information based on vehicle data. This model can model the vehicle's driving scenario as a fully vectorized representation, thereby utilizing vectorized vehicle motion information and mapping elements to plan and constrain the vehicle's driving trajectory, improving the safety of the output initial trajectory; it also avoids computationally intensive raster representation and manually designed post-processing steps, thus improving the efficiency of outputting the initial trajectory.

[0071] The planning model can be an end-to-end autonomous driving algorithm framework (Vectorized Autonomous Driving, VAD) based on vectorized scene representation, which can be applied to autonomous driving tasks such as predicting vehicle trajectories.

[0072] The VAD framework may include a first encoder, a second encoder, a third encoder, and a fourth encoder.

[0073] The first encoder is used to extract dynamic target information in the scene where the vehicle is located and output the vectorized motion features of the vehicle, which can realize dynamic target detection and vectorized trajectory prediction.

[0074] The second encoder is used to encode the input image data and convert it into a feature map from a bird's-eye view.

[0075] The third encoder is used to extract vectorized static element information in the scene, such as lane lines, curbs, and sidewalks.

[0076] The fourth encoder is used to extract information related to driving decision-making and planning based on vectorized motion features, feature maps from a bird's-eye view, and static element information, and to complete the trajectory planning of autonomous vehicles.

[0077] Specifically, the input data of the first encoder is multi-frame, multi-view image data. Based on the first encoder, the features of the image data are encoded, and a set of bird's-eye view query instructions are used to project the image features onto the bird's-eye view features to obtain the vectorized motion features of the vehicle.

[0078] Specifically, the second encoder uses a set of surrogate query instructions and map query instructions to learn vectorized feature data of each entity in the scene where the vehicle is located. The surrogate query instructions predict the trajectory of entities in the scene, outputting a one-dimensional matrix of Na×Nk×Tf×2, where Na represents the number of entities in the scene, Nk represents the driving state, Tf represents the future timestamp, and 2 represents the vehicle's coordinates from the bird's-eye view. The map query instructions take the features from the bird's-eye view as input and output vectorized instance features from the scene, which is a one-dimensional matrix of Nm×Np×2, where Nm represents the number of objects in the scene, Np is the number of points occupied by an object, and 2 represents the entity's position from the bird's-eye view.

[0079] Specifically, the third encoder uses a self-querying instruction to learn static element information in the scenario.

[0080] Specifically, the fourth encoder outputs the initial planned trajectory based on the vehicle's query instructions, motion characteristics, and static element information.

[0081] In one embodiment of this application, in order to improve the accuracy of the initial planning trajectory generated by the planning model, three vectorized planning constraints are introduced during the training of the planning model to improve the security of the planning results.

[0082] The planning constraints include those for avoiding collisions between the vehicle and other vehicles. Specifically, based on the collision constraints of the vectorized trajectories predicted by other dynamic targets in the scene and the planned trajectory of the vehicle, the vehicle's safety boundary is decomposed into lateral and longitudinal safety distances. When the distance between the planned trajectory and the predicted trajectory of another vehicle in any direction is less than a specified safety threshold, the confidence of the trajectory is reduced to avoid the intersection of the planned trajectory of the vehicle and the predicted trajectory of another vehicle.

[0083] The planning constraints also include boundary crossing constraints between the vehicle and the road boundary. Specifically, based on the predicted vectorized boundary lines, the planned trajectory of the vehicle is constrained to always remain within the drivable area. When the planned trajectory of the vehicle crosses the road boundary line, the confidence level of that trajectory is reduced.

[0084] The planning constraints also include constraints between the vehicle and the road direction. Specifically, based on the predicted road direction, the vehicle's planned trajectory orientation is constrained to maintain consistency with the road's forward direction. When there is a significant difference between the vehicle's planned trajectory orientation and the road's forward direction, the confidence level of that trajectory is reduced.

[0085] In one embodiment of this application, the image data and the vehicle data are input into a planning model to obtain the initial trajectory of the vehicle. (See [link to relevant documentation]). Figure 4 The corresponding detailed explanation.

[0086] S23, using a preset optimizer, the initial trajectory is optimized based on the driving behavior and preset traffic rule information to obtain the optimized target trajectory and control parameters.

[0087] In one embodiment of this application, the initial trajectory output by the planning model can be optimized based on the driving behavior output by the large model, thereby improving the accuracy of predicting the vehicle's driving trajectory. Specifically, the initial trajectory can be optimized using a preset optimizer based on the driving behavior and preset traffic rule information to obtain the optimized target trajectory and control parameters. This ensures that the optimized target trajectory conforms to the environmental constraints of the area where the vehicle is located and also complies with traffic rule constraints, thus improving the rationality of the target trajectory.

[0088] To evaluate the accuracy of the target trajectory obtained after optimizing the initial trajectory, the optimizer's performance is assessed based on information from multiple dimensions. The optimizer's optimization objectives include both general optimization objectives and generative model optimization objectives. General optimization objectives include general speed costs and general acceleration costs. The generative model optimization objective ensures that the planned path is adjusted to a certain extent based on the planning results of the larger model. The generative model influences the final trajectory formation through fuzzy choices made by logical reasoning regarding speed control and steering control. The generative model's objectives include generative speed costs, generative steering costs, generative following costs, and generative acceleration costs.

[0089] Among them, when evaluating the performance of the vehicle speed optimizer, it can calculate the difference between the vehicle's speed and the current road segment's speed limit, as well as the speed difference between the vehicle's speed and other vehicles in the area where the vehicle is located.

[0090] Among them, based on the differences between the control parameters of vehicle driving (e.g., differences in acceleration and steering angle), it is possible to evaluate the magnitude of the change in vehicle driving state when the vehicle is driving along the trajectory output by the optimizer.

[0091] In one embodiment of this application, the optimized target trajectory may include multiple point data, each point data including a timestamp, location information, velocity information, and acceleration information. For example, the target trajectory may take the form of: {(t0, x0, y0, v0, a0), (t1, x1, y1, v1, a1), ..., (t... n x n y n v n a n )}, where t0 represents the first point when the vehicle travels along the target trajectory, x0 and y0 represent the position information of the vehicle when it is at the first point of the target trajectory, v0 represents the velocity of the vehicle when it is at the first point of the target trajectory, and a0 represents the acceleration of the target when it is at the first point.

[0092] The control parameters output by the optimizer may include parameters for controlling vehicle acceleration, parameters for controlling vehicle steering, parameters for controlling vehicle deceleration, etc., and this application does not limit them.

[0093] In one embodiment of this application, the method for training the optimizer can be found in [reference needed]. Figure 5 , Figure 6 , Figure 7 and Figure 8 The corresponding detailed explanation.

[0094] In one embodiment of this application, the initial trajectory is updated using a preset optimizer based on the driving behavior and pre-stored traffic rule information. For details on the method for obtaining the updated target trajectory and control parameters, please refer to [link to relevant documentation]. Figure 9 The corresponding detailed explanation.

[0095] S24, based on the control parameters, control the vehicle to move along the target trajectory.

[0096] In one embodiment of this application, the driving state of a vehicle can be controlled based on control parameters, thereby controlling the vehicle to move along a target trajectory. For example, the vehicle can be accelerated based on acceleration parameters in the control parameters, thereby controlling the vehicle to move along the target trajectory; the vehicle can also be steered or changed lanes based on steering parameters in the control parameters, thereby controlling the vehicle to remain within the target trajectory. This ensures the compliance and security of the final optimized target trajectory and improves the accuracy and traceability of controlling the vehicle to move along the target trajectory based on control parameters.

[0097] As can be seen from the above technical solutions, the embodiments of this application extract environmental features based on image data and generative models, which can ensure the integrity of environmental information and provide data support for driving trajectory decision-making. These environmental features, including distance, dynamics, and scene association, can directly provide understandable and usable input for driving behavior decision-making, avoiding information gaps between perceived information and decision-making needs. Furthermore, the generative model has context modeling capabilities, which can cope with extreme scenarios where traditional perception is prone to failure, improving the credibility and stability of environmental features. In addition, determining driving behavior based on environmental features, vehicle data, and session data ensures that the generated driving behavior conforms to the collaborative adaptation between the user, vehicle, and environment, preventing the subsequently generated target trajectory and control parameters from deviating from actual user decisions. Moreover, determining driving behavior based on environmental information constraints avoids a disconnect between decision-making and vehicle physical capabilities, reducing the probability of dangerous operations. Predicting the initial trajectory based on image data and vehicle data, and optimizing the initial trajectory based on driving behavior and traffic rules, ensures the real-time performance and reliability of trajectory planning through a coarse-filtering planning and fine-tuning optimization mechanism. This ensures that trajectory planning is synchronized with real-time road condition changes, avoiding the risk of accidents caused by missed lane-changing opportunities due to computational delays. It can ensure the compliance and safety of the final optimized target trajectory, and improve the accuracy and traceability of controlling the vehicle to move along the target trajectory based on control parameters.

[0098] like Figure 3 The diagram shown is a flowchart of a method for determining the driving behavior of a vehicle according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the driving behavior of a vehicle according to an embodiment of this application includes the following steps.

[0099] S30: The environmental features, vehicle data, and session data are formatted according to the logical chain information in the preset prompt words to obtain prompt information.

[0100] In one embodiment of this application, environmental features, vehicle data, and session data can be formatted based on prompt words containing logical chain information to obtain prompt information. This ensures that the generative model can understand the information in the environmental features, vehicle data, and session data based on the prompt information, thereby improving the accuracy of the predicted vehicle driving behavior. The driving behavior information can be a collection of multiple vehicle behaviors described in natural language, and the various pieces of information in the driving behavior have a temporal correlation.

[0101] For example, the prompt message may include: Background information: You are driving on a 4-lane road; You are currently driving in the leftmost lane; Your current location is (0.00, 0.00), your speed is 24.03 meters per second, your acceleration is -0.05 meters per second squared, and there are currently 3 lanes available. You can turn left or right.

[0102] There are other vehicles around your vehicle. Here is some basic information about them: Vehicle 1 is traveling in the lane to your right and is in front of you; Vehicle 1 is currently traveling at (278, 96), and this position is to the right front.

[0103] Vehicle 2 is traveling in the same lane as you and is in front of you; Vehicle 2 stops at (222, 0.00), and this position is behind.

[0104] Your optional actions include: Maintain current speed and stay in the current lane. Behavior indicator: 1; Turn right, changing to the right lane of the current lane, behavior sign: 2; Accelerate, control vehicle acceleration, behavior indicator: 3; Decelerate, control the vehicle to slow down, behavior indicator: 4.

[0105] Please output the corresponding driving behavior based on the following content {}.

[0106] S31, Based on the prompt information, the corresponding driving behavior is determined using the generative model.

[0107] In one embodiment of this application, a prompt message can be input into a generative model to obtain the driving behavior corresponding to the prompt message. The driving behavior is used to indicate the driving behavior predicted by the generative model that is consistent with the vehicle's environment.

[0108] For example, the results output by the generative model may include "acceleration, left turn, no overtaking, shortening following distance, and increasing acceleration".

[0109] like Figure 4 The diagram shown is a flowchart of a driving control method according to an embodiment of this application. The order of the steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The driving control method provided in this embodiment includes the following steps.

[0110] S40, the first encoder in the planning model is used to extract features from the state information in the vehicle data to obtain the vectorized motion features of the vehicle.

[0111] Specifically, the input data of the first encoder is multi-frame, multi-view image data. Based on the first encoder, the features of the image data are encoded, and a set of bird's-eye view query instructions are used to project the image features onto the bird's-eye view features to obtain the vectorized motion features of the vehicle.

[0112] S41, the second encoder in the planning model is used to extract features from the image data to obtain the top-view features of the vehicle.

[0113] Specifically, the second encoder uses a set of surrogate query instructions and map query instructions to learn vectorized feature data of each entity in the scene where the vehicle is located. The surrogate query instructions predict the trajectory of entities in the scene, outputting a one-dimensional matrix of Na×Nk×Tf×2, where Na represents the number of entities in the scene, Nk represents the driving state, Tf represents the future timestamp, and 2 represents the vehicle's coordinates from the bird's-eye view. The map query instructions take the features from the bird's-eye view as input and output vectorized instance features from the scene, which is a one-dimensional matrix of Nm×Np×2, where Nm represents the number of objects in the scene, Np is the number of points occupied by an object, and 2 represents the entity's position from the bird's-eye view.

[0114] S42, the third encoder in the planning model is used to extract features from the map information in the vehicle data to obtain the map features of the area where the vehicle is located.

[0115] Specifically, the third encoder uses a self-querying instruction to learn static element information in the scenario.

[0116] S43, based on the motion characteristics, the top-down characteristics, and the map characteristics, the initial trajectory of the vehicle is predicted using the fourth encoder in the planning model.

[0117] Specifically, the fourth encoder outputs the initial planned trajectory based on the vehicle's query instructions, motion characteristics, and static element information.

[0118] like Figure 5 The diagram shown is a flowchart of a training optimizer method according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The training optimizer method provided in this embodiment includes the following steps.

[0119] S50, based on the historical image data of the vehicle, the historical environmental characteristics of the vehicle are determined using the generative model.

[0120] In one embodiment of this application, image data of the vehicle's surroundings captured by an in-vehicle camera can be analyzed to determine the vehicle's environmental characteristics, and the vehicle's driving trajectory and control parameters can be predicted based on these environmental characteristics, thereby improving the accuracy of the generated vehicle driving trajectory and control parameters.

[0121] Specifically, the environmental features of the vehicle can be determined using a pre-defined generative model based on the vehicle's image data. These environmental features indicate information about entities surrounding the vehicle.

[0122] For example, the environmental features of a vehicle output by a generative model may include: <Type: Vehicle, Status: Driving, Visual Description: Red Sedan, Location: [12,13,11,12], Image: Front, Importance: High>; <Type: Traffic light, Status: null, Visual description: Red light for left turn, green light for straight line, Location: [12,13,11,12], Image: Front, Importance: High>; <Type: Traffic sign, Status: null, Visual description: Speed ​​limit 80km / h, Location: [12,13,11,12], Image: Front left, Importance: High>.

[0123] For example, the environmental features of the vehicle output by the generative model can also include the weather conditions, lane conditions and emergency situations in the area where the vehicle is located. For example, the environmental features can specifically include: <sunny daytime, high visibility>; <water accumulation ahead>; <intersection, construction zone, severe congestion>; <three lanes in total, the left lane is passable, the right lane is not passable>.

[0124] S51, input the historical environmental features, the vehicle's historical vehicle data, and the historical session data into the generative model to obtain the vehicle's behavior data.

[0125] In one embodiment of this application, in order to predict the driving behavior of a vehicle in the future period based on the environmental information of the area where the vehicle is located, the driving behavior of the vehicle can be determined by using a generative model based on environmental characteristics, vehicle data and pre-acquired session data.

[0126] Among them, driving behavior refers to the actions that control the driving state of the vehicle. For example, driving behavior may include turning left, turning right, overtaking, increasing acceleration, etc.

[0127] In one embodiment of this application, environmental features, vehicle data, and session data can be formatted based on prompt words containing logical chain information to obtain prompt information. This ensures that the generative model can understand the information in the environmental features, vehicle data, and session data based on the prompt information, thereby improving the accuracy of the predicted vehicle driving behavior. The driving behavior information can be a collection of multiple vehicle behaviors described in natural language, and the various pieces of information in the driving behavior have a temporal correlation.

[0128] For example, the results output by the generative model may include "acceleration, left turn, no overtaking, shortening following distance, and increasing acceleration".

[0129] S52, input the historical image data and the historical vehicle data into the planning model to obtain the trajectory data of the vehicle.

[0130] In one embodiment of this application, in order to initially determine the initial trajectory of a vehicle, a preset planning model can be used to analyze the features related to vehicle movement in image data and vehicle data, and output the initial trajectory of the vehicle.

[0131] The image data can be image data of the vehicle's surroundings captured by the vehicle's onboard camera. Specifically, the image data of the vehicle's surroundings captured by the onboard camera can include image data of the front of the vehicle, the rear of the vehicle, the front left of the vehicle, the rear left of the vehicle, the front right of the vehicle, and the rear right of the vehicle.

[0132] Vehicle data can include vehicle status information, environmental information, and navigation information.

[0133] Specifically, environmental information can include map information and traffic reference lines for the area where the vehicle is located. Vehicle status information can include the vehicle's location (e.g., latitude and longitude), the vehicle's orientation, and its real-time speed. Vehicle navigation information can include target reference lines.

[0134] In one embodiment of this application, the planning model can be a vehicle data processing framework, including multiple encoders and decoders, for outputting vectorized trajectory information based on vehicle data. This model can model the vehicle's driving scenario as a fully vectorized representation, thereby utilizing vectorized vehicle motion information and mapping elements to plan and constrain the vehicle's driving trajectory, improving the safety of the output initial trajectory; it also avoids computationally intensive raster representation and manually designed post-processing steps, thus improving the efficiency of outputting the initial trajectory.

[0135] S53, using a pre-built initial optimizer, the trajectory data is updated based on the behavior data and the traffic rule information to obtain the updated predicted trajectory and prediction parameters.

[0136] In one embodiment of this application, the initial trajectory output by the planning model can be optimized based on the driving behavior output by the large model, thereby improving the accuracy of predicting the vehicle's driving trajectory. Specifically, the initial trajectory can be optimized using a preset optimizer based on the driving behavior and preset traffic rule information to obtain the optimized target trajectory and control parameters. This ensures that the optimized target trajectory conforms to the environmental constraints of the area where the vehicle is located and also complies with traffic rule constraints, thus improving the rationality of the target trajectory.

[0137] S54, based on the behavioral data, the trajectory data, the historical environmental features, the predicted trajectory, and the prediction parameters, determine the loss value of the initial optimizer.

[0138] To evaluate the accuracy of the target trajectory obtained after optimizing the initial trajectory, the optimizer's performance is assessed based on information from multiple dimensions. The optimizer's optimization objectives include both general optimization objectives and generative model optimization objectives. General optimization objectives include general speed costs and general acceleration costs. The generative model optimization objective ensures that the planned path is adjusted to a certain extent based on the planning results of the larger model. The generative model influences the final trajectory formation through fuzzy choices made by logical reasoning regarding speed control and steering control. The generative model's objectives include generative speed costs, generative steering costs, generative following costs, and generative acceleration costs.

[0139] Among them, when evaluating the performance of the vehicle speed optimizer, it can calculate the difference between the vehicle's speed and the current road segment's speed limit, as well as the speed difference between the vehicle's speed and other vehicles in the area where the vehicle is located.

[0140] Among them, based on the differences between the control parameters of vehicle driving (e.g., differences in acceleration and steering angle), it is possible to evaluate the magnitude of the change in vehicle driving state when the vehicle is driving along the trajectory output by the optimizer.

[0141] S55, update the initial optimizer according to the loss value, and stop updating the initial optimizer when the loss value meets the preset first termination condition, so as to obtain an optimizer trained to the convergence state.

[0142] In one embodiment of this application, the initial optimizer can be updated based on the loss value to improve its performance. The preset first termination condition can be that the loss value is less than or equal to a preset first termination threshold.

[0143] Specifically, if the loss value is greater than the preset first termination threshold, it indicates a large difference between the predicted trajectory output by the optimizer and the trajectory data. In this case, the optimizer can be updated to improve the accuracy of the output. If the loss value is less than or equal to the preset first termination threshold, it indicates a small difference between the predicted trajectory output by the optimizer and the trajectory data. In this case, updating the optimizer can be stopped, resulting in an optimizer that has been trained to a convergent state.

[0144] like Figure 6 The diagram shown is a flowchart of a method for determining the loss value of an optimizer according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the loss value of an optimizer provided in this embodiment includes the following steps.

[0145] S60, based on the trajectory data, the prediction parameters, the predicted trajectory, the historical environmental features, and the traffic rule information, determine the first loss of the initial optimizer.

[0146] In one embodiment of this application, the degree of conformity between the predicted speed in the prediction parameters and the speed limit in the traffic rule information can be used to characterize the degree of conformity between the vehicle driving based on the prediction parameters and the traffic rule information.

[0147] Specifically, the system can determine the first position of any entity around the vehicle in the coordinate system corresponding to the vehicle based on historical environmental characteristics, determine the second position of the vehicle in the coordinate system based on the predicted trajectory, and then determine the probability of the entity around the vehicle colliding with the vehicle based on the difference between the first position and the second position.

[0148] Specifically, the vehicle's dynamic loss can be determined based on the acceleration, rate of change of acceleration, steering angle, and rate of change of steering angle in the predicted parameters. A greater dynamic loss indicates a greater difference between the vehicle's actual driving state and the predicted state when the vehicle is controlled according to the predicted parameters.

[0149] Specifically, lane loss can be determined based on lane position information indicated by historical environmental characteristics and the difference between the second position. Furthermore, the first loss of the initial optimizer is determined based on speed loss, position loss, dynamic loss, and lane loss.

[0150] S61, based on the behavioral data, the prediction parameters, the prediction trajectory, and the historical environmental features, determine the second loss of the initial optimizer.

[0151] In one embodiment of this application, a second speed loss can be determined based on the speed change data of the vehicle indicated by the behavioral data, and according to the difference between the predicted speed and the speed of any entity around the vehicle indicated by the historical environmental features. Furthermore, based on the vehicle's steering state indicated by the behavioral data, and according to the difference between the first position and the second position, a steering loss is determined when controlling the vehicle to travel along a predicted trajectory according to predicted parameters.

[0152] In one embodiment of this application, the second loss of the initial optimizer can also be determined based on the second velocity loss and the steering loss. Specifically, the second velocity loss and the steering loss can be weighted and summed to obtain the second loss of the initial optimizer. This application does not limit the specific method for determining the second loss.

[0153] S62, perform a weighted summation operation on the first loss and the second loss to obtain the loss value of the initial optimizer.

[0154] In one embodiment of this application, a weighted summation operation can be performed on the first loss and the second loss to obtain the loss value of the initial optimizer. This application does not limit the specific method for determining the weights of the first loss value and the second loss value.

[0155] like Figure 7 The diagram shown is a flowchart of a method for determining the first loss value of an optimizer according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the first loss value of an optimizer provided in this embodiment includes the following steps.

[0156] S70, based on the difference between the predicted speed in the prediction parameters and the speed limit in the traffic rule information, a first speed loss is determined.

[0157] In one embodiment of this application, a first speed loss can be determined based on the difference between the predicted speed in the prediction parameters and the speed limit in the traffic rule information. A larger first speed loss indicates a greater difference between the vehicle's driving state and the traffic rule constraints during the process of controlling the vehicle to travel along the predicted trajectory according to the prediction parameters.

[0158] The calculation method for the first velocity loss satisfies the following relationship: ; in, This represents the first velocity loss; The predicted speed of the vehicle at time t is indicated by the prediction parameters. This represents the speed limit specified in traffic rules.

[0159] S71, based on the historical environmental characteristics, determine the first position of any entity around the vehicle in the coordinate system corresponding to the vehicle; and, based on the predicted trajectory, determine the second position of the vehicle in the coordinate system.

[0160] In one embodiment of this application, in order to determine the accuracy of the prediction parameters and the prediction trajectory, the first position of any entity around the vehicle in the coordinate system corresponding to the vehicle can be determined based on historical environmental characteristics, and the second position of the vehicle in the coordinate system can be determined based on the prediction trajectory.

[0161] For example, when the entity surrounding the vehicle is a traffic stop line, the first position indicates the position information of the stop line in the coordinate system corresponding to the vehicle; when the entity surrounding the vehicle is another vehicle, the first position indicates the position information of the other vehicle in the coordinate system.

[0162] S72, determine the position loss based on the difference between the first position and the second position.

[0163] In one embodiment of this application, the greater the difference between the first position and the second position, the closer the vehicle is to other entities in the area, the lower the probability of a collision between the vehicle and other entities, and the smaller the position loss; conversely, the smaller the difference between the first position and the second position, the farther the vehicle is from other entities in the area, the higher the probability of a collision between the vehicle and other entities, and the greater the position loss.

[0164] For example, when the entity surrounding the vehicle is a traffic stop line, the specific method for determining the position loss satisfies the following relationship: ; in, Position loss when the representative entity is a traffic stop line; The second position of the vehicle at time t, as indicated by the predicted trajectory; When the representative entity is a traffic stop line, it represents the position of that entity in the coordinate system corresponding to the vehicle.

[0165] For example, when the entities surrounding the vehicle are other vehicles, the specific method for determining the location loss satisfies the following relationship: ; in, Position loss when the representative entity is other vehicles around the vehicle; The second position of the vehicle at time t, as indicated by the predicted trajectory; This represents the position of the entity with index i around the vehicle in the coordinate system corresponding to the vehicle.

[0166] S73, determine the dynamic loss based on the acceleration, the rate of change of the acceleration, the steering angle, and the rate of change of the steering angle in the prediction parameters.

[0167] In one embodiment of this application, the dynamic loss of the vehicle can also be determined based on the acceleration, the rate of change of acceleration, the steering angle, and the rate of change of steering angle in the prediction parameters.

[0168] The greater the acceleration, the greater the change in the vehicle's driving state when the vehicle is controlled to travel along the predicted trajectory according to the predicted parameters. In this case, the lower the accuracy of the predicted parameters and the predicted trajectory, the higher the dynamic loss of the vehicle.

[0169] The greater the rate of change of acceleration, the greater the magnitude of the change in the vehicle's driving state when the vehicle is controlled to travel along the predicted trajectory according to the predicted parameters. In this case, the accuracy of the predicted parameters and the predicted trajectory is lower, and the dynamic loss of the vehicle is higher.

[0170] The larger the steering angle, the greater the change in the vehicle's driving state when the vehicle is controlled to travel along the predicted trajectory according to the predicted parameters. In this case, the accuracy of the predicted parameters and the predicted trajectory is lower, and the dynamic loss of the vehicle is higher.

[0171] The greater the rate of change of the steering angle, the greater the change in the vehicle's driving state when the vehicle is controlled to travel along the predicted trajectory according to the predicted parameters. In this case, the accuracy of the predicted parameters and the predicted trajectory is lower, and the dynamic loss of the vehicle is higher.

[0172] The greater the dynamic loss, the greater the difference between the vehicle's driving state and its actual driving state when the vehicle is controlled according to the predicted parameters.

[0173] S74, based on the lane position information indicated by the historical environmental features and the difference between the second position, determine the lane loss.

[0174] In one embodiment of this application, the greater the difference between the second position and the lane position information indicated by historical environmental features, the higher the deviation between the vehicle's predicted trajectory and the lane position during the process of controlling the vehicle to travel along the predicted trajectory according to the predicted parameters, and the higher the lane loss.

[0175] The specific method for determining lane loss satisfies the following relationship: ; in, Represents lane loss; This represents the position information of the centerline of the lane line in the coordinate system corresponding to the vehicle. This represents the vehicle's location information at time t, as indicated by the predicted data.

[0176] S75, based on the speed loss, the position loss, the dynamic loss, and the lane loss, determine the first loss of the initial optimizer.

[0177] In one embodiment of this application, the first loss of the initial optimizer can be obtained by weighted summation based on speed loss, position loss, dynamic loss, and lane loss. A higher first loss indicates lower accuracy of the predicted parameters and trajectory output by the initial optimizer.

[0178] like Figure 8 The diagram shown is a flowchart of a method for determining the second loss value of an optimizer according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the second loss value of an optimizer provided in this embodiment includes the following steps.

[0179] S80, based on the speed change data of the vehicle indicated by the behavioral data, a second speed loss is determined according to the difference between the predicted speed and the speed of any entity around the vehicle indicated by the historical environmental features.

[0180] In one embodiment of this application, when a vehicle is traveling based on speed change data indicated by behavioral data, if the difference between the predicted speed and the speed of any entity around the vehicle indicated by historical environmental characteristics is higher, it indicates that the relative position change between the vehicle and the surrounding entities is greater, and the stability of the vehicle's driving state is lower. Therefore, the probability of a collision between the vehicle and the surrounding entities is higher, and the second speed loss of the vehicle is higher.

[0181] The method for determining the second velocity loss satisfies the following relationship: ; in, This represents the second velocity loss; This represents the predicted speed of other vehicles at time t; The vehicle's speed at time t, as indicated by the predicted data; The generative model indicates that the vehicle is decelerating. The generative model indicates that the vehicle is in an acceleration state.

[0182] S81, based on the vehicle's steering state indicated by the behavioral data, determine the steering loss according to the difference between the first position and the second position.

[0183] In one embodiment of this application, when a vehicle is driving according to a steering state indicated by behavioral data, if the difference between the first position of other entities and the second position of the vehicle itself is smaller, it indicates that the probability of a collision between the vehicle and other entities is higher, and the steering loss of the vehicle is greater.

[0184] The method for determining the steering loss satisfies the following relationship: ; in, Represents a loss due to a shift in strategy; This represents the predicted steering angular velocity of other vehicles at time t; The angular velocity of the vehicle at time t, as indicated by the predicted data; The generative model indicates that the vehicle is in a left-turning state. The generative model indicates that the vehicle is in a right-turning state.

[0185] S82, determine the second loss of the initial optimizer based on the second speed loss and the steering loss.

[0186] In one embodiment of this application, a weighted summation operation can be performed based on the second speed loss and the steering loss to obtain the second loss of the initial optimizer. A higher second loss indicates a higher probability of collision with other entities in the vehicle's area while the vehicle travels along the predicted trajectory according to the predicted parameters output by the initial optimizer; consequently, the accuracy of the predicted parameters and trajectory output by the initial optimizer is lower.

[0187] like Figure 9 The diagram shown is a flowchart of a method for determining an updated target trajectory and control parameters according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining an updated target trajectory and control parameters provided in this embodiment includes the following steps.

[0188] S90, based on the driving behavior and pre-stored traffic rule information, the optimizer is used to predict the predicted position of the vehicle.

[0189] In one embodiment of this application, the vehicle electronic device can control the driving state of the vehicle based on the driving behavior output by the generative model. Therefore, based on the driving behavior and pre-stored traffic rule information, it can predict the predicted position of the vehicle after driving according to the driving state indicated by the driving behavior.

[0190] S91, the initial trajectory is updated based on the deviation between the initial trajectory and the predicted position until the deviation meets a preset second termination condition, and the target trajectory and the control parameters are obtained based on the updated initial trajectory.

[0191] In one embodiment of this application, the second termination condition may be that the deviation between any point in the initial trajectory and the predicted position is less than or equal to a preset second termination threshold.

[0192] Specifically, if the deviation between the predicted position and the point in the initial trajectory is greater than the preset second termination threshold, it indicates that the vehicle will deviate from the predicted position after traveling along the initial trajectory, and the accuracy of the initial trajectory is low, so the initial trajectory can be updated.

[0193] Specifically, when the deviation between the predicted position and the point in the initial trajectory is less than or equal to the preset second termination threshold, it indicates that the deviation between the vehicle and the predicted position after traveling along the initial trajectory is small, and the accuracy of the initial trajectory is high. Therefore, the updated initial trajectory can be determined as the target trajectory, and the prediction parameters corresponding to the predicted position can be determined as control parameters.

[0194] Please see Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement a driving control method as described in any of the above embodiments.

[0195] In one embodiment of this application, the electronic device 100 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a driving control program.

[0196] Figure 10 Only an electronic device 100 with memory 12 and processor 13 is shown; those skilled in the art will understand that... Figure 10 The structure shown does not constitute a limitation on the electronic device 100, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0197] Combination Figure 2The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the driving control method. The processor 13 can execute the plurality of instructions to achieve the following: inputting image data collected by the vehicle into a generative model to obtain entity information around the vehicle; inputting the entity information, vehicle data, and pre-acquired session data into the generative model to obtain the driving behavior of the vehicle; inputting the image data and vehicle data into a planning model to obtain the initial trajectory of the vehicle; using a preset optimizer, optimizing the initial trajectory according to the driving behavior and preset traffic rule information to obtain the optimized target trajectory and control parameters; and controlling the vehicle to move along the target trajectory based on the control parameters.

[0198] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0199] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 100 may also include input / output devices, network access devices, etc.

[0200] It should be noted that electronic device 100 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0201] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a portable hard drive of the electronic device 100. In other embodiments, the memory 12 can be an external storage device of the electronic device 100, such as a plug-in portable hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device 100. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of a driving control program, but also to temporarily store data that has been output or will be output.

[0202] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 100, connecting to various components of the electronic device 100 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., driving control programs) and calls data stored in the memory 12 to perform various functions and process data for the electronic device 100.

[0203] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the various driving control method embodiments described above, for example... Figure 2 The steps are shown.

[0204] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 100.

[0205] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the driving control method described in the various embodiments of this application.

[0206] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0207] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0208] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0209] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 10 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0210] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions that are executed by a processor in an electronic device to implement the driving control method described in any of the above embodiments.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0213] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A driving control method, characterized in that, The method includes: The image data collected by the vehicle is input into the generative model to obtain entity information around the vehicle. The entity information, the vehicle data, and the pre-acquired session data are input into the generative model to obtain the vehicle's driving behavior. The image data and vehicle data are input into the planning model to obtain the initial trajectory of the vehicle; Using a preset optimizer, the initial trajectory is optimized based on the driving behavior and preset traffic rule information to obtain the optimized target trajectory and control parameters; Based on the control parameters, the vehicle is controlled to move along the target trajectory.

2. The driving control method according to claim 1, characterized in that, The method further includes training the optimizer, including: Based on the historical image data of the vehicle, the historical environmental characteristics of the vehicle are determined using the generative model. The historical environmental features, the vehicle's historical vehicle data, and the historical session data are input into the generative model to obtain the vehicle's behavioral data. The historical image data and the historical vehicle data are input into the planning model to obtain the vehicle's trajectory data; Using a pre-built initial optimizer, the trajectory data is updated based on the behavioral data and the traffic rule information to obtain the updated predicted trajectory and prediction parameters; Based on the behavioral data, the trajectory data, the historical environmental features, the predicted trajectory, and the prediction parameters, the loss value of the initial optimizer is determined; The initial optimizer is updated based on the loss value, and the update of the initial optimizer is stopped when the loss value meets a preset first termination condition, thus obtaining an optimizer trained to a convergent state.

3. The driving control method according to claim 2, characterized in that, The step of determining the loss value of the initial optimizer based on the behavioral data, the trajectory data, the historical environmental features, the predicted trajectory, and the prediction parameters includes: Based on the trajectory data, the prediction parameters, the predicted trajectory, the historical environmental features, and the traffic rule information, the first loss of the initial optimizer is determined; The second loss of the initial optimizer is determined based on the behavioral data, the prediction parameters, the prediction trajectory, and the historical environmental characteristics. The first loss and the second loss are weighted and summed to obtain the loss value of the initial optimizer.

4. The driving control method according to claim 3, characterized in that, The step of determining the first loss of the initial optimizer based on the trajectory data, the prediction parameters, and the traffic rule information includes: The first speed loss is determined based on the difference between the predicted speed in the prediction parameters and the speed limit in the traffic rule information. Based on the historical environmental characteristics, determine the first position of any entity around the vehicle in the coordinate system corresponding to the vehicle; and, based on the predicted trajectory, determine the second position of the vehicle in the coordinate system. Based on the difference between the first position and the second position, determine the position loss; Based on the acceleration, the rate of change of the acceleration, the steering angle, and the rate of change of the steering angle in the predicted parameters, the dynamic loss is determined; Based on the lane location information indicated by the historical environmental features and the difference between the second location, lane loss is determined; Based on the speed loss, the position loss, the dynamic loss, and the lane loss, the first loss of the initial optimizer is determined.

5. The driving control method according to claim 4, characterized in that, The step of determining the second loss of the initial optimizer based on the behavioral data, the prediction parameters, the prediction trajectory, and the historical environmental features includes: Based on the vehicle's speed change data indicated by the behavioral data, a second speed loss is determined according to the difference between the predicted speed and the speed of any entity around the vehicle indicated by the historical environmental features. Based on the vehicle's steering state indicated by the behavioral data, a steering loss is determined according to the difference between the first position and the second position; The second loss of the initial optimizer is determined based on the second velocity loss and the steering loss.

6. The driving control method according to claim 1, characterized in that, The step of determining the vehicle's driving behavior using the generative model based on the environmental characteristics, the vehicle's vehicle data, and pre-acquired session data includes: The environmental features, vehicle data, and session data are formatted based on the logical chain information in the preset prompt words to obtain prompt information; Based on the prompt information, the corresponding driving behavior is determined using the generative model.

7. The driving control method according to claim 1, characterized in that, The step of inputting the image data and the vehicle data into the planning model to obtain the initial trajectory of the vehicle includes: The first encoder in the planning model is used to extract features from the state information in the vehicle data to obtain the vectorized motion features of the vehicle. The second encoder in the planning model is used to extract features from the image data to obtain the top-view features of the vehicle. The third encoder in the planning model is used to extract features from the map information in the vehicle data to obtain the map features of the area where the vehicle is located. Based on the motion characteristics, the top-down characteristics, and the map characteristics, the initial trajectory of the vehicle is predicted using the fourth encoder in the planning model.

8. The driving control method according to any one of claims 1 to 7, characterized in that, The step involves updating the initial trajectory using a preset optimizer based on the driving behavior and pre-stored traffic rule information to obtain the updated target trajectory and control parameters, including: Based on the driving behavior and pre-stored traffic rule information, the optimizer is used to predict the predicted position of the vehicle. The initial trajectory is updated based on the deviation between the initial trajectory and the predicted position until the deviation meets a preset second termination condition. The target trajectory and the control parameters are then obtained based on the updated initial trajectory.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the driving control method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the driving control method as described in any one of claims 1 to 8.

Images