Automatic driving motion planning method and system driven by large language model

Abstract

The invention discloses an automatic driving motion planning method and system driven by a large language model, and the method comprises the steps: obtaining a current vehicle state based on a perception function library, and obtaining observation information; acquiring recall information based on the observation information; obtaining cognitive information based on the recall information; providing cognitive information to the large language model, generating an advanced driving strategy combination, and planning a driving track according to a reasoning result and the advanced driving strategy combination; the driving track is arranged in an analog simulation environment to be executed, and feedback is obtained; and selecting to circularly execute the driving track based on the feedback result, or reflecting the driving track. The method has the beneficial effects that the data generalization ability is improved, lifetime learning is realized, the collaboration flexibility and expandability of the system are improved, and the adaptability and reliability of the system in the real world are enhanced.

Description

Technical Field

[0001] The invention relates to the field of autonomous driving technology, and specifically to an autonomous driving motion planning system driven by a large language model. Background Art

[0002] Autonomous driving technology has made significant progress in recent years and is widely used in scenarios such as parks, intersections, and highways. Existing autonomous driving systems usually rely on deep neural networks and large-scale datasets such as nuScenes and Waymo, which are carefully curated and rich in diversity, promoting research and development in this field.

[0003] The current mainstream motion planning algorithms require rule design based on a large amount of data, or learning from data based on deep learning. These methods perform well on standard datasets and have a certain positive effect when facing private and diverse real-world datasets.

[0004] However, the limitations of existing technologies are also worth noting, including: 1. Insufficient generalization ability: Existing motion planning methods perform poorly on private and diverse real-world datasets. These datasets may contain new variables such as uncommon road shapes, different driving norms, and pedestrian behaviors, making it difficult for traditional methods to adapt.

[0005] 2. Lack of flexibility: Traditional methods lack flexibility and cannot make adaptive adjustments in unknown scenarios, requiring reconfiguration and model training.

[0006] 3. Poor interpretability: Existing methods lack interpretability and cannot provide a basis for decision-making based on human knowledge. This makes it difficult for the system to make reasonable decisions in complex driving scenarios. Summary of the invention

[0007] The purpose of the invention is to provide a lifelong learning autonomous driving motion planning method and system driven by a large language model, which utilizes the human knowledge and learning ability provided by the large language model to achieve adaptive motion planning for diverse and private data sets through perception, memory, reflection and reasoning modules; by developing the Dream-env simulation environment, the system can be tested and optimized in a closed-loop environment, and achieve lifelong learning and experience generalization to solve the problems raised in the above-mentioned background technology.

[0008] To achieve the above purpose, the invention provides the following technical solutions: In a first aspect, the present invention provides a large language model driven autonomous driving motion planning method, the method comprising the following steps: Get the current vehicle state based on the perception library, and get observation information based on the order of vehicle state, detection, and road; Based on the observation information, choose whether to perform recall to obtain the historical trajectory of surrounding vehicles or the road information that has passed but is now out of sight, and obtain the recall information; Based on the recall information, the decision moment of each current scene information is embedded into the vector by the text vector model, and the vector is used to make inferences in the vector database of experience memory, match the most similar driving scene, and obtain cognitive information; Provide cognitive information to the large language model to generate a combination of advanced driving strategies, and plan the driving trajectory based on the reasoning results and the combination of advanced driving strategies; Arrange the driving trajectory in a simulation environment to execute and obtain feedback; Based on the feedback results, the driving trajectory is selected to be executed repeatedly, or the driving trajectory is corrected.

[0009] Preferably, the current vehicle state is obtained based on the perception function library, and observation information is obtained according to the sequence of vehicle state, detection and road, including: Force the execution of the function in the perception function library once to obtain the vehicle's position coordinates, speed, orientation angle, acceleration, wheel angle, vehicle length and width information, and obtain the current task and the target coordinate point of the current task; Based on the vehicle status and task description, select one of the functions to call to obtain the required information; The required information is integrated as query keys through multiple rounds of conversations.

[0010] Preferably, the method selects to call one of the functions based on the vehicle state and the task description to obtain the required information, and the perception function includes a state class function, a detection class function and a road class function.

[0011] Preferably, the step of selecting whether to recall and obtain the previous historical tracks of surrounding vehicles or the road information that has been driven before but is now out of sight based on the observation information, and obtaining the recall information includes: Based on the recall function, the historical trajectory of the object with the specified ID in the previous k seconds is obtained; Based on the recall class, the road information traveled in the previous k seconds is obtained.

[0012] Preferably, the step of obtaining the decision moment when each current scene information is embedded into a vector by a text vector model based on the recall information, performing reasoning in a vector database of experience memory through the vector, matching the most similar driving scene, and obtaining cognitive information includes: The scene information obtained by observation and recall is stored in the vector memory using the vector embedding generated by the large language model as the key; Arrange the keys according to the cosine similarity from large to small to get the most relevant memory; Choose a small sample memory to assist in reasoning and planning later; Obtain previous lessons learned and common traffic knowledge from the database.

[0013] Preferably, the cosine similarity calculation formula is: in, is the th vector embedded in the current scene value, It is the first vector in the vector library that needs to be queried. value.

[0014] Preferably, the advanced driving strategy combination includes: Discrete driving behaviors: stop, go forward, turn left, reverse left, turn right, reverse right; Speed ​​estimation: constant speed, deceleration, rapid deceleration, deceleration to zero, acceleration, rapid acceleration.

[0015] Preferably, the step of selecting to cyclically execute the driving trajectory based on the feedback result, or reflecting on the driving trajectory, comprises: Display danger or collision information in search feedback information; Analyze the steps where errors occur based on an explainable chain of thought combined with the reasoning process and simulation environment; Generate a new driving trajectory and mark the wrong steps; Store the marked error steps.

[0016] In a second aspect, the present invention further provides an autonomous driving motion planning system, the system comprising: An observation module, the observation module is used to execute a state-based function to obtain observation information through the state of the vehicle, the detection and the order of the road; A recall module, wherein the recall module selects whether to recall the historical trajectory of the vehicle or the road information that has been driven but is now out of sight according to the observation information, and then obtains the recall information; A cognitive module is used to take the recall information and embed each current scene information into the vector decision moment by the text vector model, and use the vector to make inferences in the vector database of experience memory, match the most similar driving scene, and obtain cognitive information; A planning module, the planning module is used to provide cognitive information to the large language model, generate a high-level driving strategy combination, and plan a driving trajectory based on the reasoning result and the high-level driving strategy combination; A simulation environment module, wherein the simulation environment module is used to execute a driving trajectory and obtain feedback; The reflection module selects to cyclically execute the driving trajectory or to correct the driving trajectory based on the feedback results.

[0017] In a third aspect, the present invention also provides an electronic device, comprising at least one processor, wherein the processor is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0018] Compared with the prior art, the invention has the following beneficial effects: 1. The present invention improves data diversity and generalization capabilities for difficult scenarios: The present invention uses a data set containing 225 scenarios, covering a variety of complex driving environments such as parks, intersections and highways, and significantly improves the system's generalization capabilities in different scenarios.

[0019] 2. The core engine of the present invention is based on a large language model (LLM), which has human knowledge and learnability. Through continuous data and scene generation, the system can continuously update and expand the memory experience library, thereby achieving lifelong learning and adapting to new driving environments and challenges.

[0020] 3. The present invention includes modules such as observation, recollection, reflection and cognition. The modules work together to improve the decision-making accuracy and security of the system. The modular design makes the system highly flexible and scalable, which is convenient for adjustment and optimization according to actual needs.

[0021] 4. The present invention develops the Dream-Env simulation environment, which provides a closed-loop environment and can automatically generate and visualize scenes based on data sets; the simulation environment not only improves the testing efficiency of the system, but also enhances the adaptability and reliability of the system in the real world.

[0022] Through the above advantages, the autonomous driving motion planning system based on lifelong learning driven by a large model of the present invention has demonstrated significant technological progress in improving data diversity, generalization ability and flexibility, and provides an innovative and effective solution for the application of autonomous driving motion planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a large language model-driven autonomous driving motion planning method according to an embodiment of the present invention; Figure 2 for Figure 1 The sub-flow chart of S100; Figure 3 for Figure 1 The sub-flow chart of S200; Figure 4 for Figure 1 The sub-flow chart of S300; Figure 5 for Figure 1The sub-flow chart of S600; Figure 6 A schematic diagram of a vehicle in a simulation environment according to an embodiment of the present invention; Figure 7 A module diagram of an autonomous driving motion planning system provided by an embodiment of the present invention; Figure 8 A flow chart of a specific embodiment provided by the present invention; Fig. 9 This is a module diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the invention to clearly and completely describe the technical solutions in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the invention.

[0025] See also Figure 1 The invention provides a technical solution: a large language model driven autonomous driving motion planning method, the method comprising the following steps: S100: obtaining the current vehicle state based on the perception function library, and obtaining observation information according to the sequence of vehicle state, detection and road; S200: Based on the observation information, select whether to perform recall to obtain the historical trajectory of surrounding vehicles or the road information that has passed but is now out of the field of view, and obtain the recall information; S300: Based on the recall information, the decision moment when each current scene information is embedded into the vector by the text vector model is obtained, and reasoning is performed in the vector database of experience memory through the vector to match the most similar driving scene and obtain cognitive information; S400: providing cognitive information to a large language model, generating an advanced driving strategy combination, and planning a driving trajectory based on the reasoning result and the advanced driving strategy combination; S500: Arranging the driving trajectory in a simulation environment for execution to obtain feedback; S600: Select to execute the driving trajectory in a loop based on the feedback result, or to correct the driving trajectory.

[0026] See also Figure 2 , S100 specifically includes the steps of: S110: Force the execution of a function in the perception function library to obtain the vehicle's position coordinates, speed, orientation angle, acceleration, wheel angle, vehicle length and width information, and obtain the current task and the target coordinate point of the current task.

[0027] Specifically, before starting this embodiment, radars and sensors are installed on the vehicle to collect real-time road and other object behavior data, including road length, type, vehicle speed, object coordinates, and driving direction. The collected data will be stored and added to the data set for use in subsequent autonomous driving scenarios.

[0028] In this embodiment, the vehicle used for implementation is equipped with various sensors and radars for obtaining information about road conditions, vehicle status and surrounding objects in real time. The vehicle is equipped with an intelligent agent, which can be, for example, ADAS, etc. It directly retrieves information from the environment and provides the current vehicle status to the large language model. The intelligent agent can autonomously choose to execute various tool functions as needed to convert the current scene data into descriptive text. In order to make it easier to call the large language model, we integrate various perception functions into a function library, which includes three types of functions: status, detection and road, covering functions of different usages and their descriptions. The structure of the function library can be added or deleted according to the actual situation of the application scenario, and has a customizable feature.

[0029] Specifically, in this embodiment, the state class function is used (Extracting ego vehicle input) uses sensors to obtain the vehicle’s own state, such as speed and heading angle; Specifically, in this embodiment, the detection function can be used to detect the state of objects within the detection range, such as object category, steering angle, position and other information. For example: By calling (Get the detection result of the object in front) Information about objects in the 16*15m space around the function, including the type and various states of the objects; By calling (Get the detection results of surrounding objects) The function obtains the object information in the surrounding space, including object type and various states; By calling (Get the detection results of objects within the range) The function obtains the object information within the space range according to the parameters given by the intelligent agent, including the object type and various states.

[0030] Specifically, in this embodiment, the road class function can obtain information about the traffic road within the field of view of the vehicle, such as the position of the shoulder and the center line of the lane.

[0031] For example, by calling The (get nearest shoulder) function gets the information of the nearest k roads, including the road length, starting point coordinates and end point coordinates; By calling (Get the distance to the shoulder at a specific location) The function obtains the distance information to the two nearest shoulders, as well as the starting and ending coordinates of the shoulders.

[0032] S120: Based on the vehicle status and task description, select and call one of the functions to obtain the required information.

[0033] Specifically, the agent obtains the description of this type of function according to the state information of the vehicle and the task description, according to the order of state, detection and road, and chooses to call one of the functions to obtain the required information.

[0034] S130: Integrate the required information as a query key through multiple rounds of conversations.

[0035] See also Figure 3 , step S200 further includes: S210: Obtain the historical running trajectory of the object with the specified ID in the previous k seconds based on the recall function; S220: Acquire the road information traveled in the previous k seconds based on the recall class.

[0036] Specifically, in this embodiment, the agent calls the recall function to choose whether to recall the historical trajectories of surrounding vehicles or the road information that has been passed but is now out of sight according to the information obtained from the observed scene.

[0037] The recall functions include: (Get the historical trajectory of an object) function is used to get the historical trajectory of an object with a specified id in the previous k seconds; (Obtain observations outside the field of view) function, used to obtain the road information traveled in the previous k seconds.

[0038] In this embodiment, this embodiment is responsible for recalling information from previous driving scenes to help the system make more accurate reasoning and decisions in complex scenes. In real-world driving scenes, drivers can assist in reasoning by recalling previous scene information. Letting the large language model recall and use previously observed information helps the large language model make more accurate reasoning and decisions in complex scenes, thereby improving the performance of the autonomous driving system.

[0039] See also Figure 4 The specific reasoning process of step S300 includes: S310: storing the scene information obtained by observing the information and recalling the information in the vector memory using the vector generated by the large language model as a key; S320: sort the keys according to cosine similarity from large to small to get the most relevant memories, the cosine similarity calculation formula is: in, is the vector of the current scene embedding values, It is the first vector in the vector library that needs to be queried. values; S330: Select a small sample memory to be used as auxiliary reasoning and planning; S340: Obtain previous experience and common traffic knowledge from the database.

[0040] In this embodiment, the intelligent agent will then obtain previous lessons learned and traffic common sense knowledge from the database, including traffic light rules, driving regulations, etc., and then integrate the selected experience memories, lessons learned and traffic common sense knowledge from the observation module for use by the planning module.

[0041] Specifically, common sense traffic rules include: Keep a safe distance: Keep a safe distance from the vehicle in front of you to prevent emergency braking.

[0042] Overtake with caution: When overtaking, you must make a prompt decision and do not hesitate or delay. Make sure it is safe before overtaking.

[0043] Obey traffic rules: strictly follow traffic signals and signs to ensure safe driving.

[0044] Be aware of your blind spots: Before changing lanes or turning, carefully check your blind spots to make sure there are no other vehicles or pedestrians.

[0045] Smooth acceleration and deceleration: Gentle acceleration and deceleration reduces impact on the vehicle and improves fuel efficiency.

[0046] Use brakes correctly: When you need to slow down or stop, apply the brakes in advance to avoid danger caused by sudden braking.

[0047] Turning skills: Slow down when turning, keep the speed stable and ensure the stability of the vehicle.

[0048] In this embodiment, driving common sense, lessons learned, and memories are stored, allowing the system to search for similar experience memories at each driving moment. Human drivers use their common sense to plan their driving behavior, such as complying with local traffic regulations, and refer to past experience when encountering similar scenarios; at the same time, they will learn lessons from accidents and avoid making similar mistakes again. In order to enable the large language model to have these capabilities, we use retrieval enhancement generation technology and design cognitive modules to store driving common sense, lessons learned, and experience memories. For each driving moment, the large language model can use scene information as a key to search for similar past experience memories. Common sense memory: encapsulates the basic knowledge of drivers for safe driving on the road, such as local traffic regulations and necessary driving skills. Lessons learned: Contains a series of summaries and lessons generated after dangerous situations in previous driving scenarios were corrected. Experience memory: Experience memory contains a series of past driving scenes at different times, where each scene consists of scene information, decision reasoning at the time, and motion planning.

[0049] In a specific embodiment of the invention, the intelligent agent retrieves 20 relevant experience memories based on the scene information, allows the intelligent agent to select 10 for reference, and provides the intelligent agent with traffic common sense and lessons learned.

[0050] In an embodiment of the present invention, the advanced driving strategy combination in step S400 is defined as a combination of discrete driving behaviors and speed estimation. For example, the advanced driving plan of decelerating and driving straight is generated by combining and combining the straight driving and decelerating. Then, the intelligent body combines the reasoning process and the advanced driving strategy to generate a trajectory simulating the human driving mode according to the scene information.

[0051] Specifically, discrete driving behaviors: stop, go forward, turn left, reverse left, turn right, reverse right; speed estimation: constant speed, deceleration, rapid deceleration, deceleration to zero, acceleration, rapid acceleration.

[0052] Subsequently, the unknown driving trajectory is executed in the simulation environment to obtain feedback. Then, the scene information available to the ego vehicle at the current time is obtained from the simulation, and this information is used as the input of the observation module, such as Figure 6 As shown, in this embodiment, the simulation environment in step S500 uses Dream-Env to generate a visual schematic diagram. Dream-Env can read the map and vehicle information in the data and construct a map at the beginning of the task, including roads and obstacles, to provide a global scene display, where yellow represents the vehicle of the embodiment of the present invention, and blue represents other objects, such as other vehicles, bicycles and pedestrians.

[0053] See also Figure 5 , step S600 specifically includes: S610: Searching for danger or collision information displayed in feedback information; S620: Analyze the steps where errors occur based on an explainable thinking chain combined with the reasoning process and the simulation environment; S630: Generate a new driving trajectory and mark the wrong steps; S640: Store the marked erroneous steps.

[0054] In this embodiment, the driving trajectory completed by motion planning is input into the simulation environment for execution and feedback is obtained. If the feedback shows that no collision or dangerous event occurs, the entire motion planning process is executed in a loop.

[0055] On the contrary, if a dangerous situation occurs, such as colliding with other vehicles or driving on the shoulder of the road, it means that the agent has made an unreasonable motion plan. Due to the interpretable thought chain process, the agent can find the cause of failure from the previous reasoning process and the feedback given by the simulation environment, re-plan the motion to obtain a more accurate trajectory, and summarize the lessons learned from the mistakes. These lessons and experience memories will also be stored in the memory bank of cognitive information.

[0056] See also Figure 7 In an embodiment of the present invention, there is also provided an automatic driving motion planning system, which can execute any of the methods described in the above embodiments and has the function of an intelligent agent in the above embodiments, and the system includes: An observation module 100 is used to execute state-based functions and obtain observation information through the state of the vehicle, the detection and the order of the road; A recall module 200, wherein the recall module selects whether to recall the historical trajectory of the vehicle or the road information that has been driven but is now out of sight according to the observation information, and then obtains the recall information; A cognitive module 300, which is used to obtain the decision moment when each current scene information is embedded into the vector by the text vector model from the recall information, and to perform reasoning in the vector database of experience memory through the vector to match the most similar driving scene and obtain cognitive information; A planning module 400, which is used to provide cognitive information to the large language model, generate a high-level driving strategy combination, and plan a driving trajectory based on the reasoning result and the high-level driving strategy combination; A simulation environment module 500, which is used to execute a driving trajectory and obtain feedback; The reflection module 600 selects to cyclically execute the driving trajectory or to reflect on the driving trajectory based on the feedback result.

[0057] In an embodiment of the present invention, the above-mentioned large language model uses Qwen2-72B-Instruct fine-tuned by instructions as a basic large language model of an autonomous driving motion planning system. It should be noted that the large language model can also be replaced by other large language models.

[0058] In the embodiment of the present invention, the observation module 100 has a built-in function library, which includes a state function, a detection function and a road function; the recall module 200 sets the longest time that can be recalled to 64 moments.

[0059] In an embodiment of the present invention, the cognitive module 300 obtains the decision moment of vector embedding through the large language model. The initial experience library contains 100 artificial experiences. At each driving moment, the system is set to select 10 from the experience library as small sample learning cases, and allows the experience library to continue to accumulate.

[0060] In an embodiment of the present invention, the simulation environment module 500 sets the decision frequency to 1 Hz, which is consistent with the acquisition frequency of the data set and the decision frequency of the driver in reality, and sets a timeout time for each scene to the total duration of the scene plus 20.

[0061] The working principle of the system of the present invention is: Observation scenario: First, the agent will be forced to execute at least one state class function to obtain some current self-vehicle state, then according to the order of state, detection and road, the agent will obtain the description of this class function and choose to call one of them to collect the required information. If necessary, the large language model needs to provide the parameters of the execution function during the call. Finally, through multiple rounds of dialogue, the agent will eventually get the required scene information after integration.

[0062] Recall scenario: The intelligent body will choose whether to recall the historical trajectories of surrounding vehicles or the roads that have been driven before but are now out of sight based on the state information obtained from the observation scenario. This information will be integrated with the information obtained from the observation scenario as a key to query relevant small sample experience in the cognitive module.

[0063] Cognitive memory: At each decision moment, the current scene information will be embedded into a vector by the text vector model. This vector is used to search in the vector database of experience memory to find the most similar driving scene. These driving scenes are provided to the large language model, which is then screened based on the similarity of each driving scene. These experience memories are provided to the large language model in the form of small samples to help it make accurate inferences and decisions about the current scene. If the large language model makes the correct inferences and decisions, or obtains a revised reasoning process through rethinking, the text vector model will embed the scene information into a vector, pair it with the reasoning process and decision, and store it in the experience memory vector library.

[0064] Planning and execution: First, let the large language model generate a high-level driving strategy, which is defined as a combination of discrete driving behaviors and speed estimates. For example, a combination of straight driving and deceleration generates a high-level driving plan of decelerating and going straight. The purpose of motion planning is to generate a safe and efficient driving trajectory. Each trajectory is defined as an array of a series of position coordinate points and the steering angle of the vehicle. The large language model will generate trajectories that simulate human driving patterns based on scene information, reasoning process, and high-level driving strategies. Finally, we will input these trajectories into the simulation environment for execution.

[0065] Experience accumulation and feedback: After the motion plan is input into the simulation environment and executed, feedback will be obtained. If the feedback shows that no collision or dangerous events have occurred, the entire motion planning process will be executed in a loop. On the contrary, if a dangerous situation occurs, such as a collision with other vehicles or driving on the shoulder of the road, this indicates that the agent has made an unreasonable motion plan. Due to the interpretable chain of thought process, the large language model can find the cause of failure from the previous reasoning process and the feedback given by the simulation environment, re-plan the motion to obtain a more accurate trajectory, and summarize experience from the mistakes. These experiences will also be stored in the memory bank.

[0066] Through the above mechanism, the method of the present invention collects data and evaluates it in 141 real scenarios. Since the goal of the present invention is to allow the large language model to reach the target point as quickly as possible without danger. Therefore, the present invention uses failure rate, timeout rate and average time consumption as evaluation indicators. If the large language model can safely reach the target point without timeout, it is considered a success, and the time taken to reach the target point is the time consumption. When calculating the average time consumption of all scenarios, the time consumption of the failure scenario is regarded as its maximum planning time.

[0067] Table 1: Experimental results It can be seen from Table 1 that by gradually removing some modules from the complete modules and the complete method provided by the present invention, the failure rate and the timeout rate are gradually increased.

[0068] Therefore, the autonomous driving motion planning system of the present invention can effectively improve driving safety and efficiency and adapt to diverse real-world driving scenarios.

[0069] See also Figure 8 In a specific embodiment of the present invention, for example, the specific process of a lane change and passing case in a park, the agent first calls the detect_surrounding_objects function of the detection to obtain information about surrounding objects, as follows: object ID: 33, object type: vehicle, position: (-76.91, 124.17), heading angle: 2.96, speed: 0.01, acceleration: -0.03, ... In a specific embodiment of the present invention, the agent then pays attention to the vehicle with ID 33 based on the surrounding object information, so it chooses to obtain its four-second historical trajectory to assist in subsequent reasoning and planning. The information is as follows: The historical trajectory of object 33 (last 4 seconds): (-76.83,124.17) (-76.83,124.17) (-76.87,124.19) (-76.90,124.21), ... In a specific embodiment of the present invention, the agent performs motion planning based on scene information, experience memory and system prompts, and generates a reasonable motion trajectory for the simulation environment to simulate and execute. The information is as follows: [text], driving plan: turn right at a constant speed. Planned trajectory and heading: [(-70.0591,128.7131,1.1178), (-70.2524,129.0527,1.2236), (-70.5366,129.5676,1.3239), (-70.9043,130.2745,1.4178)] In a specific embodiment of the present invention, the agent re-reasons and reflects based on the feedback execution results, summarizes the failure experience and gives a more reasonable trajectory. The information is as follows: How to avoid such errors in the future: [lessons learned], planning process: [text], ..., [output] See also Fig. 9 , Fig. 9 A schematic diagram of an electronic device 20 that can implement an embodiment of the present invention is shown, and the electronic device is intended to represent various forms of control devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0070] The electronic device 20 includes at least one processor 21, and a memory connected to the at least one processor 21, such as a read-only memory (ROM) 22, a random access memory (RAM) 23, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 22 or the computer program loaded from the storage unit 28 to the random access memory (RAM) 13. In the RAM 23, various programs and data required for the operation of the electronic device 20 can also be stored. The processor 21, ROM 22 and RAM 23 are connected to each other through a bus 24. An input / output (I / O) interface 25 is also connected to the bus 24.

[0071] A number of components in the electronic device 20 are connected to the I / O interface 25, including: an input unit 26, such as a keyboard, a mouse, etc.; an output unit 27, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 29, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 29 allows the electronic device 20 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0072] The processor 21 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 21 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 21 performs the various methods and processes described above.

[0073] In some embodiments, the method of the above embodiment may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 28. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 20 via the ROM 22 and / or the communication unit 29. When the computer program is loaded into the RAM 23 and executed by the processor 21, one or more steps in the method described above may be performed. Alternatively, in other embodiments, the processor 21 may be configured to perform the method of the above embodiment in any other appropriate manner (e.g., by means of firmware).

[0074] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0076] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0078] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0079] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0080] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0081] Although embodiments of the invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A large language model driven autonomous driving motion planning method, characterized in that: The steps include: Get the current vehicle status based on the perception function library, and obtain observation information based on the order of vehicle status, detection and road; Based on the observation information, choose whether to perform recall to obtain the historical trajectory of surrounding vehicles or the road information that has passed but is now out of sight, and obtain the recall information; Based on the recall information, the decision moment of each current scene information is embedded into the vector by the text vector model, and the vector is used to make inferences in the vector database of experience memory, match the most similar driving scene, and obtain cognitive information; Provide cognitive information to the large language model to generate a combination of advanced driving strategies, and plan the driving trajectory based on the reasoning results and the combination of advanced driving strategies; Arrange the driving trajectory in a simulation environment to execute and obtain feedback; Based on the feedback results, the driving trajectory is selected to be executed repeatedly, or the driving trajectory is corrected.

2. The method for autonomous driving motion planning driven by a large language model according to claim 1, characterized in that: The perception function library is used to obtain the current state of the vehicle, and the observation information is obtained according to the sequence of vehicle state, detection and road, including: Force the execution of the function in the perception function library once to obtain the vehicle's position coordinates, speed, orientation angle, acceleration, wheel angle, vehicle length and width information, and obtain the current task and the target coordinate point of the current task; Based on the vehicle status and task description, select one of the functions to call to obtain the required information; The required information is integrated as query keys through multiple rounds of conversations.

3. The large language model driven autonomous driving motion planning system according to claim 2, characterized in that: The perception function includes a state-type function, a detection-type function and a road-type function.

4. The method for autonomous driving motion planning driven by a large language model according to claim 3, characterized in that: Based on the observation information, selecting whether to perform recall to obtain the previous historical tracks of surrounding vehicles or the road information that has been driven before but is now out of the field of vision, and obtaining the recall information includes: Based on the recall function, the historical trajectory of the object with the specified ID in the previous k seconds is obtained; Based on the recall class, the road information traveled in the previous k seconds is obtained.

5. The method for autonomous driving motion planning driven by a large language model according to claim 4, characterized in that: The method of obtaining the decision moment when each current scene information is embedded into the vector by the text vector model based on the recall information, performing reasoning in the vector database of experience memory through the vector, matching the most similar driving scene, and obtaining cognitive information includes: The scene information obtained by observation and recall is stored in the vector memory using the vector embedding generated by the large language model as the key; The keys are sorted from large to small according to the cosine similarity to get the most relevant a memory; choose A small sample memory is used as a later aid for reasoning and planning; Obtain previous lessons learned and common traffic knowledge from the database.

6. The method for autonomous driving motion planning driven by a large language model according to claim 5, characterized in that: The cosine similarity calculation formula is: in, is the vector of the current scene embedding values, It is the first vector in the vector library that needs to be queried. value.

7. The method for autonomous driving motion planning driven by a large language model according to claim 6, characterized in that: The advanced driving strategy combination includes: Discrete driving behaviors: stop, go forward, turn left, reverse left, turn right, reverse right; Speed ​​estimation: constant speed, deceleration, rapid deceleration, deceleration to zero, acceleration, rapid acceleration.

8. The method for autonomous driving motion planning driven by a large language model according to claim 7, characterized in that: The step of selecting to cyclically execute the driving trajectory based on the feedback result, or reflecting on the driving trajectory, includes: Display danger or collision information in search feedback information; Analyze the steps where errors occur based on an explainable chain of thought combined with the reasoning process and simulation environment; Generate a new driving trajectory and mark the wrong steps; Store the marked error steps.

9. An autonomous driving motion planning system, characterized in that: The system includes: An observation module, the observation module is used to execute a state-based function to obtain observation information through the state of the vehicle, the detection and the order of the road; A recall module, wherein the recall module selects whether to recall the historical trajectory of the vehicle or the road information that has been driven but is now out of sight according to the observation information, and then obtains the recall information; A cognitive module is used to take the recall information and embed each current scene information into the vector decision moment by the text vector model, and use the vector to make inferences in the vector database of experience memory, match the most similar driving scene, and obtain cognitive information; A planning module, the planning module is used to provide cognitive information to the large language model, generate a high-level driving strategy combination, and plan a driving trajectory based on the reasoning result and the high-level driving strategy combination; A simulation environment module, wherein the simulation environment module is used to execute a driving trajectory and obtain feedback; The reflection module selects to cyclically execute the driving trajectory or to correct the driving trajectory based on the feedback results.

10. An electronic device, characterized in that: The method comprises at least one processor, wherein the processor is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

Images