Automatic driving planning method based on visual language large model and related equipment

Abstract

The invention discloses an automatic driving planning method based on a visual language large model and related equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring driving scene information; the driving scene information is input into a visual language large model, automatic driving planning parameters output by the visual language large model are obtained, and the visual language large model is obtained through sample driving scene information and thinking chain fine adjustment; the thinking chain is used for assisting the visual language large model to perform multi-stage reasoning analysis on the sample driving scene information. By implementing the technical scheme provided by the invention, the reliability of model output is improved through multi-stage reasoning, and the safety of the automatic driving system is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an autonomous driving planning method based on a large visual language model and related equipment. Background Art

[0002] With the rapid development of autonomous driving technology, how to achieve efficient and safe autonomous driving planning has become a hot topic in current research. In complex and changing road environments, autonomous driving systems need to quickly understand driving scenarios and make appropriate driving decisions, which places high demands on autonomous driving planning algorithms.

[0003] Currently, most autonomous driving planning methods rely mainly on artificial rules and traditional machine learning algorithms. These methods usually use predefined rules or models to analyze driving scenarios and then generate corresponding driving plans based on the analysis results.

[0004] However, these methods still have obvious shortcomings when dealing with complex and dynamically changing driving scenes. Especially after the introduction of the visual language big model, the visual language big model may produce visual hallucinations, that is, erroneously understand elements or situations that do not actually exist when analyzing driving scenes. Such visual hallucinations may cause the autonomous driving system to make wrong judgments, seriously affecting driving safety. Summary of the invention

[0005] The present invention provides an automatic driving planning method and related equipment based on a large visual language model, which improves the reliability of the model output through multi-stage reasoning and enhances the safety of the automatic driving system.

[0006] In a first aspect of the present invention, a method for autonomous driving planning based on a visual language large model is provided, comprising: Obtain driving scene information; The driving scene information is input into the visual language big model, and the autonomous driving planning parameters output by the visual language big model are obtained, wherein the visual language big model is obtained by fine-tuning the sample driving scene information and the thinking chain, and the thinking chain is used to assist the visual language big model in performing multi-stage reasoning analysis on the sample driving scene information.

[0007] In a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an autonomous driving planning method based on a large visual language model as described above is implemented.

[0008] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the automatic driving planning method based on the visual language large model as described in any one of the above is implemented.

[0009] In a fourth aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned autonomous driving planning methods based on a large visual language model.

[0010] In summary, one or more technical solutions provided in the present invention have at least the following technical effects or advantages: By inputting the driving scene information into the visual language large model that has been fine-tuned by the thinking chain, the autonomous driving planning parameters suitable for the current driving scene can be obtained. This method has significant advantages over the traditional methods that rely on predefined rules or models in the background technology. First, the visual language large model that has been fine-tuned by the thinking chain can perform multi-stage reasoning analysis on the driving scene, which enables the model to understand the complex and changeable driving environment more comprehensively and deeply, thereby generating planning parameters that are more suitable for the current scene. Secondly, the introduction of the visual language large model enables the system to have stronger semantic understanding and contextual reasoning capabilities, and can better handle dynamically changing driving scenes, overcoming the shortcomings of traditional methods in complex scenes. In addition, the introduction of the thinking chain helps to alleviate the visual illusion problem that may be caused by the visual language large model, and improves the reliability of the model output through multi-stage reasoning, thereby enhancing the safety of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is a flowchart of an autonomous driving planning method based on a large visual language model provided by an embodiment of the present invention.

[0013] Figure 2 It is a bird's-eye view schematic diagram provided by an embodiment of the present invention.

[0014] Figure 3 It is a schematic diagram of a lane map provided by an embodiment of the present invention.

[0015] Figure 4 It is a schematic diagram of a fine-tuning process of a large visual language model provided by an embodiment of the present invention.

[0016] Figure 5 It is a schematic diagram of a fine-tuning process of a large visual language model provided by an embodiment of the present invention.

[0017] Figure 6 This is a simulation schematic diagram provided by an embodiment of the present invention.

[0018] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Please refer to Figure 1 , Figure 1 This is a flowchart of an automatic driving planning method based on a large visual language model provided by an embodiment of the present invention. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, or can be run on an automatic driving planning system based on a large visual language model based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the method may include the following steps: S101. Acquire driving scene information.

[0021] Among them, driving scene information refers to a comprehensive and structured description of the current environment of the autonomous driving vehicle, including but not limited to road layout, location and status of traffic participants, traffic rules and other related information. In the embodiment of the present application, it can be understood as a highly abstract and structured scene representation obtained after being collected and processed by multiple sensors, mainly including two parts: one is a bird's-eye view of the scene, and the other is the corresponding text description.

[0022] Furthermore, driving scene information is used to provide comprehensive and easy-to-process input about the current driving environment to the visual language model. Complex three-dimensional scene information can be converted into two-dimensional planes and text forms, which greatly reduces the complexity of the data and facilitates the processing and understanding of the visual language model. Secondly, by combining the two modalities of image and text, it is possible to capture all aspects of the scene more comprehensively, including both intuitive spatial relationships and semantic information that is difficult to visualize. Thirdly, this representation method retains key information in the scene while filtering out unnecessary details, allowing the model to focus on elements that are critical to safe driving. Finally, this standardized scene representation method helps to improve the generalization ability of the model, enabling it to better cope with various complex driving environments.

[0023] Based on the above embodiment, as an optional embodiment, in S101: the step of obtaining driving scene information may further include the following steps: S201: Acquire a driving scene.

[0024] Among them, the driving scene refers to the overall environment in which the autonomous driving vehicle is located during driving, including road structure, traffic participants, traffic facilities, weather conditions and other external factors that may affect driving decisions. In the embodiment of the present application, it can be understood as a set of raw data collected in real time by a variety of sensor systems carried by the autonomous driving vehicle, which together constitute a comprehensive description of the current driving environment. Specifically, the driving scene covers the road environment (such as lane lines, road conditions, road geometry, etc.), traffic participants (such as other vehicles, pedestrians, cyclists and their dynamic information), traffic facilities (such as traffic lights, traffic signs, roadblocks, etc.), weather and lighting conditions, and special circumstances (such as traffic accidents, road construction, emergency vehicles, etc.). This diversified scene definition ensures that the autonomous driving system can fully perceive and understand the complex driving environment.

[0025] S202: Acquire road elements in the driving scene.

[0026] Among them, road elements refer to the basic components that constitute the driving scene, which are structured and semantic representations of complex driving environments. These elements include but are not limited to lanes, traffic participants, traffic facilities and other key factors that can affect autonomous driving decisions. In the embodiments of this application, it can be understood as entities with specific attributes and semantics extracted and identified from the original driving scene data, which are mainly divided into three categories: lanes, dynamic objects and static objects.

[0027] Lane elements include lane lines, lane types, lane widths, and other information. Dynamic object elements mainly refer to other vehicles, pedestrians, cyclists, etc., and include their dynamic features such as position, speed, and direction of movement. Static object elements include objects that do not move or rarely move, such as traffic signs, traffic lights, and roadblocks, and record their static features such as type and position. Each road element is given clear semantic information and attributes, allowing the system to accurately understand their role and impact in the driving scenario.

[0028] S203: Obtain a bird's-eye view composed of road elements.

[0029] Please refer to Figure 2 , Figure 2 It is a bird's-eye view schematic diagram provided by the present invention. Specifically, the bird's-eye view refers to a two-dimensional image representation method for observing and representing the driving scene from a high altitude perspective. This representation method projects the three-dimensional real-world scene onto a plane, centered on the autonomous driving vehicle, and displays the layout of the surrounding environment and the spatial relationship of various road elements. In the embodiment of the present application, it can be understood as a graphical representation of various road elements such as lanes, dynamic objects, and static objects in a two-dimensional coordinate system with the vehicle as the origin.

[0030] Based on the above embodiment, as an optional embodiment, in S203: the step of obtaining a bird's-eye view composed of road elements may further include the following steps: S301. Convert a dynamic object into an arrow, wherein the arrow is used to represent the relative movement trend of the dynamic object.

[0031] In specific implementation, the system first identifies and locates dynamic objects in the scene based on sensor data, including other vehicles, pedestrians, and cyclists. Then, the motion vector of each dynamic object is calculated using the position change between consecutive frames or the speed information obtained directly from the sensor. This motion vector contains speed and direction information. Next, the position of each dynamic object in the bird's-eye view is represented by a specific icon, and an arrow is superimposed on the icon. The direction of the arrow is consistent with the direction of movement of the object, and the length of the arrow is proportional to the speed of the object.

[0032] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a lane map provided by the present invention. In order to enhance readability, arrows of different colors can be used to distinguish different types of dynamic objects, for example, red arrows are used to represent other vehicles, and blue arrows are used to represent pedestrians. In addition, the uncertainty of speed or the reliability of prediction can be indicated by the thickness or transparency of the arrows. For stationary dynamic objects (such as parked vehicles), special marks (such as dots or small circles) can be used to distinguish them.

[0033] S302: Convert the lane into a lane graph, wherein the lane graph includes nodes and edges, the nodes represent the center lines of the lanes, the edges include successor edges and adjacent edges, the successor edges represent nodes of the same lane, and the adjacent edges represent nodes of adjacent lanes.

[0034] Specifically, in the process of obtaining a bird's-eye view of road elements, converting lanes into lane graphs is a key step. Lane graphs are a structured lane network representation method that describes the topological relationship between lanes in the form of nodes and edges.

[0035] Furthermore, the nodes in the lane graph represent the center lines of the lanes, while the edges are divided into two types: successor edges and adjacent edges. Successor edges are used to connect adjacent nodes on the same lane, indicating the direction of the vehicle in that lane; adjacent edges connect adjacent nodes on different lanes, indicating the possible lane change paths of the vehicle. This representation method is important because it can simplify the complex road network into a graph structure that is easy for computers to process, thus facilitating subsequent path planning and decision making.

[0036] In specific implementation, the system first needs to extract lane information from the raw sensor data, including the type, location, and shape of the lane. Then, each lane is sampled at equal intervals to generate a series of discrete points, which will serve as nodes in the lane map. The sampling interval is usually set to a fixed value, such as 10 meters or 20 meters, to control the computational complexity while ensuring accuracy. Next, the system analyzes the relationship between adjacent nodes. If two nodes belong to the same lane, a successor edge is added between them; if two nodes belong to adjacent lanes, an adjacent edge is added. In this process, special situations such as lane bifurcation and merging need to be considered, and the connection relationship between nodes and edges should be adjusted accordingly.

[0037] S303: Construct a bird's-eye view corresponding to the arrows, static objects, and lane map.

[0038] S204: Convert the bird's-eye view into a text description to obtain driving scene information.

[0039] The purpose of converting the bird's-eye view into text description is to convert visual information into a form that can be directly processed by the language model, thereby providing a more comprehensive and structured input for the subsequent visual language model. The text description not only contains the spatial relationship information in the bird's-eye view, but also can express semantic information that is difficult to directly present in the image, such as the speed of the vehicle, the intention of the pedestrian, etc., which makes the driving scene information richer and more complete.

[0040] In specific implementation, the system will identify and classify various elements in the bird's-eye view, including lanes, dynamic objects, and static objects. Then, based on the constructed lane map, the system will describe the overall layout of the road, including the number of lanes, lane types, and the relationship between lanes. For dynamic objects, the system will record their position, speed, and direction of movement, especially their position and movement state relative to the autonomous driving vehicle. Static objects will be described as relatively fixed reference points, including their type and position. In addition, the system will analyze key information in the scene, such as potential dangerous situations, traffic rules, etc., and incorporate this information into the text description.

[0041] When generating text descriptions, the system uses predefined templates and natural language generation technology to ensure the accuracy and readability of the description. For example, for a typical urban road scene, the text description may include the following content: "The vehicle is located in the right lane of a two-way four-lane road, and the current speed is 40km / h. There is a red car traveling at a speed of 35km / h 50 meters ahead. There is a blue SUV traveling at a speed of 45km / h in the left front lane. There is a traffic light 10 meters to the right of the road, and the light is currently green. There is a pedestrian waiting to cross the road at the intersection 200 meters ahead." This description not only contains the basic information of each element in the scene, but also reflects the spatial relationship and dynamic characteristics between them.

[0042] S102. Input the driving scene information into the visual language big model to obtain the autonomous driving planning parameters output by the visual language big model, wherein the visual language big model is obtained by fine-tuning the sample driving scene information and the thinking chain, and the thinking chain is used to assist the visual language big model in performing multi-stage reasoning and analysis on the sample driving scene information.

[0043] Among them, the visual language large model refers to a large-scale neural network model that can process and understand visual information and language information at the same time. This type of model is usually based on the Transformer architecture, and after pre-training with massive multimodal data, it can demonstrate strong performance in various visual and language tasks. In the embodiment of the present application, it can be understood as a large-scale pre-trained model that is specifically optimized and fine-tuned for autonomous driving scenarios. It can receive driving scene information containing bird's-eye views and text descriptions as input, and output autonomous driving planning parameters suitable for the current scene through an internal multi-stage reasoning process.

[0044] Correspondingly, the thinking chain refers to a structured reasoning framework that breaks down a complex decision-making process into a series of logically interrelated steps or stages. This method imitates the thinking process of humans when solving problems, by breaking down a large problem into a series of small problems, and then solving them step by step to finally draw a conclusion. In the embodiment of the present application, it can be understood as a predefined, multi-stage reasoning structure, which is integrated into the visual language model to guide the model to systematically analyze and make decisions on driving scenarios.

[0045] Specifically, this chain of thinking is used to organize and structure the reasoning process of the large visual language model to ensure that the model can follow a certain logical order when dealing with complex driving scenarios. For example, a chain of thinking may include multiple stages such as scene understanding, risk assessment, behavior prediction, and decision generation, each with clear inputs and outputs. Secondly, the chain of thinking is used to improve the interpretability and traceability of the reasoning process. By recording the intermediate results of each reasoning stage, the system can clearly display the entire reasoning path from input to output, which is very important for understanding and verifying the model's decisions. Furthermore, the chain of thinking is used to enhance the model's reasoning ability, especially when dealing with complex or rare scenarios. By forcing the model to follow predefined reasoning steps, the model can be prevented from ignoring important information or making unreasonable leaps in reasoning.

[0046] In the specific implementation, the pre-processed driving scene information, including the bird's-eye view and text description, is first input into the visual language model in a predefined format. This input format usually includes the scene description, the current vehicle status, the surrounding environment information, etc. Next, the visual language model will perform multi-stage reasoning analysis based on the built-in thinking chain. The thinking chain is a structured reasoning framework that breaks down complex decision-making processes into multiple logical steps, such as scene understanding, risk assessment, behavior prediction, and decision generation. In each step, the model will generate intermediate reasoning results based on the current reasoning results and previously accumulated knowledge. These intermediate results will be used as input to the next reasoning step, thus forming a complete reasoning chain.

[0047] During the reasoning process, the visual language model will consider multiple factors, such as traffic rules, safety distance, comfort, efficiency, etc., to balance various goals and constraints. Ultimately, the model will output a set of autonomous driving planning parameters. Autonomous driving planning parameters refer to a set of instructions and constraints used to guide the behavior of autonomous driving vehicles. These parameters generally include but are not limited to target lanes, target speeds, accelerations, steering angles, safety distances, etc., which together define the expected motion state and trajectory of the vehicle in the short term. In the embodiments of the present application, it can be understood that the visual language model outputs a set of specific values ​​or instructions based on the understanding and analysis of the current driving scene, which can be directly interpreted and executed by the underlying control system.

[0048] Please refer to Figure 4 and Figure 5 , Figure 4 A schematic diagram of a fine-tuning process of a large visual language model provided by the present invention, Figure 5 The above embodiment illustrates the application process of the visual language large model. Figure 4 and Figure 5 The process of fine-tuning the large visual language model is explained.

[0049] S401: Obtain sample driving scene information.

[0050] Among them, sample driving scene information refers to a set of structured data sets used to fine-tune and evaluate large visual language models, which contain detailed descriptions of a variety of real or simulated driving environments, traffic conditions, and vehicle behaviors. In the embodiment of the present application, it can be understood as a series of standardized driving scene representations, each of which contains a bird's-eye view, a text description, and a corresponding standard driving decision or behavior label. These sample driving scene information covers a wide range of driving situations, from common daily commuting scenarios to rare emergencies, to ensure that the model can learn comprehensive driving knowledge.

[0051] S402. Determine a sample thinking chain based on the sample driving scene information, the sample thinking chain includes a first sample thinking sub-chain and a second sample thinking sub-chain, and the logical order of the first sample thinking sub-chain precedes the second sample thinking sub-chain; wherein, the first sample thinking sub-chain is used to assist the visual language big model in making middle-level decisions for the sample driving scene, and the middle-level decisions include lateral decisions and longitudinal decisions, the lateral decisions are used to characterize lane changing operations, and the longitudinal decisions are used to characterize speed control operations; wherein, the second sample thinking sub-chain is used to assist the visual language big model in determining the autonomous driving planning parameters at different stages based on the middle-level decisions.

[0052] Among them, the sample thinking chain refers to a structured reasoning framework template that simulates the thinking and decision-making process of human drivers when facing various driving scenarios. This framework decomposes complex driving decision-making tasks into a series of logically interrelated steps to guide the visual language large model to perform systematic analysis and reasoning. In the embodiment of the present application, it can be understood as a hierarchical decision structure consisting of two interrelated sub-chains, including a first sample thinking sub-chain and a second sample thinking sub-chain. The first sample thinking sub-chain is used to generate middle-level decisions, while the second sample thinking sub-chain generates specific autonomous driving planning parameters based on the middle-level decisions.

[0053] Furthermore, the first sample thinking subchain and the second sample thinking subchain are two interrelated but functionally different reasoning framework parts that constitute the sample thinking chain, which together simulate the human driving decision-making process from high-level strategies to specific operations. In the embodiment of the present application, the first sample thinking subchain can be understood as a reasoning framework that focuses on generating mid-level decisions, mainly dealing with the overall understanding of driving scenarios and macro-strategy formulation, including lateral decisions (such as whether to change lanes) and longitudinal decisions (such as the approximate direction of speed adjustment). This subchain usually includes steps such as scene perception, traffic condition assessment, safety analysis, and strategy generation, which aims to guide the model to conduct overall scene understanding and risk assessment, generate high-level driving strategies, and ensure that the model takes into account macro-level safety and efficiency when making decisions.

[0054] In contrast, the second sample thinking subchain can be understood as a reasoning framework that generates specific autonomous driving planning parameters based on mid-level decisions. It is responsible for converting the macro decisions of the first subchain into precise and executable driving instructions, which usually include path planning, speed planning, execution timing scheduling and parameter optimization. The main function of this subchain is to concretize high-level strategies, generate precise path planning and speed control instructions, optimize driving parameters to ensure comfort and energy efficiency, and formulate detailed execution plans.

[0055] Among them, the middle-level decision refers to an intermediate-level decision-making process between high-level driving strategies and low-level control instructions in the autonomous driving system. This decision-making process involves an overall evaluation of the current driving scene and the corresponding behavior selection, but has not yet been refined to specific execution parameters. In the embodiment of the present application, the middle-level decision can be understood as a set of high-level behavior instructions generated by the first sample thinking sub-chain, mainly including two aspects: horizontal decision and vertical decision.

[0056] Lateral decisions are mainly used to characterize lane-changing operations, which involve the selection and transition of vehicles between different lanes. It may include instructions such as "keep the current lane", "change lane to the left" or "change lane to the right". These decisions are based on a comprehensive analysis of factors such as the position of surrounding vehicles, lane markings, and road structure, and are intended to optimize the vehicle's driving path and position.

[0057] Longitudinal decisions are mainly used for standard speed control operations, involving the adjustment of the vehicle's speed in the current direction of travel. It may include instructions such as "accelerate", "slow down", "maintain current speed" or "emergency brake". These decisions are based on an assessment of factors such as the distance to the vehicle ahead, the road speed limit, traffic signals, etc., and are intended to ensure that the vehicle is driven in a safe, legal and efficient manner.

[0058] For example, suppose that the autonomous vehicle is driving on a three-lane urban main road and is currently in the middle lane. There is a slow-moving truck 100 meters ahead. There are many vehicles driving normally in the left lane. The right lane is relatively empty. The next intersection is 500 meters away. Based on this scenario, we can systematically determine the structure and content of the sample thinking chain.

[0059] First, the system constructs the first sample thinking subchain, which is mainly responsible for scene perception and mid-level decision making. In this subchain, the system first performs scene analysis to identify the current road type, its own position, and surrounding key elements. Next, it evaluates the potential risks of continuing to drive in the current lane and the feasibility and risk of changing lanes. At the same time, it is necessary to consider traffic rules, confirm whether the current speed meets the road speed limit, and evaluate the legality of lane changes. In addition, it is necessary to analyze the impact of different decisions on driving efficiency. Based on these comprehensive analyses, mid-level decisions are generated, including lateral decisions (such as "change lanes to the right") and longitudinal decisions (such as "maintain current speed"). This process simulates the initial judgment and decision-making process of human drivers when facing complex situations.

[0060] After the first sample thinking subchain completes the middle-level decision, it enters the second sample thinking subchain, which is responsible for converting the middle-level decision into specific planning parameters. In this subchain, path planning is first performed based on the decision of "changing lanes to the right" to determine the specific trajectory of the lane change. At the same time, according to the decision of "maintaining the current speed", combined with the lane change operation, a detailed speed curve is formulated to ensure the smoothness of the entire process. Next, the execution sequence needs to be arranged to determine the best time for the lane change operation. Then, we optimize and determine the specific operating parameters, such as steering angle, acceleration, target lane position, etc. Finally, the generated parameters are checked for safety to ensure that there will be no conflict with surrounding vehicles. This process is similar to the process of human drivers making specific plans on how to execute this decision after making a preliminary decision.

[0061] The sample thinking chain determined in the above way can effectively guide the large visual language model to gradually analyze complex driving scenarios and make reasonable decisions. This hierarchical structure not only improves the interpretability and traceability of the decision-making process, but also enhances the model's ability to handle complex scenarios.

[0062] S403: Input the sample driving scene information and the sample thinking chain into the visual language big model, and obtain the sample autonomous driving planning parameters output by the visual language big model.

[0063] In specific implementation, the system first needs to convert the sample driving scene information into a format that can be processed by the visual language model. This usually includes encoding the bird's-eye view of the scene into an image feature vector and converting the text description into a sequence of word embeddings. At the same time, the sample thought chain is encoded into a series of prompts or instructions to guide the model's reasoning process. This encoded information is combined into a structured input and then fed into the visual language model.

[0064] Inside the model, the processing follows the structure defined by the sample thinking chain. First, the model understands and analyzes the input scene information based on the first sample thinking sub-chain. This includes steps such as identifying key elements, assessing risks, considering traffic rules, and finally generating mid-level decisions. This process fully utilizes the multimodal understanding capabilities of the visual language large model, which can process image and text information simultaneously, thereby gaining a comprehensive understanding of the scene.

[0065] Next, the model further infers the specific autonomous driving planning parameters based on the generated middle-level decision according to the second sample thinking sub-chain. This process involves path planning, speed planning, execution timing scheduling, and parameter optimization. At this stage, the model needs to convert abstract decisions into specific numerical parameters, which fully utilizes the powerful reasoning and generation capabilities of the visual language large model.

[0066] Finally, the model outputs sample autonomous driving planning parameters, which may include specific values ​​such as target lane, target speed, acceleration, steering angle, etc. These output parameters not only reflect the model's understanding of the current driving scenario, but also reflect its ability to reason according to the predefined thought chain.

[0067] Based on the above embodiment, as an optional embodiment, S403 may further include the following steps: S501, input the sample driving scene information and the first sample thinking sub-chain into the visual language big model to obtain a middle-level decision.

[0068] S502. Input the second sample thinking sub-chain into the visual language big model. Through the second sample thinking sub-chain and the middle-level decision, determine the output target of the visual language big model, and restrict the input examples and output examples of the visual language big model, so as to obtain the sample autonomous driving planning parameters of the autonomous driving target in the current stage and the next stage.

[0069] Among them, the output target refers to the specific information and parameters that the visual language large model needs to generate in the autonomous driving planning task. In the embodiment of the present application, it can be understood as a set of clearly defined and structured output requirements, which specify the type, format and range of planner parameters that the model needs to generate. Specifically, the output targets include lateral control parameters (such as the center line of the lane that needs to be tracked) and longitudinal control parameters (such as speed limit, maximum acceleration and maximum deceleration). These output targets are used to guide the model to generate parameters that can be directly used in the autonomous driving control system, ensuring the practicality and executability of the model output. At the same time, clear output targets help to standardize the behavior of the model, improve the consistency and reliability of the output, and provide clear criteria for evaluating model performance.

[0070] Correspondingly, the input paradigm refers to a standardized data structure and format template used to normalize the information received by the large visual language model in the autonomous driving task. In the embodiment of the present application, it can be understood as a structured scene description format, which specifies in detail how to convert complex driving environment information into data that can be processed by the model. Specifically, the input paradigm includes the state information of the autonomous driving vehicle (such as speed, acceleration, current lane line), surrounding vehicle information (such as relative distance, speed, direction), and the selection range of planner parameters. The input paradigm is designed to ensure that the model receives comprehensive and accurate scene information, providing sufficient data support for making accurate planning decisions. At the same time, by specifying the range of parameter selection, the input paradigm also plays a role in limiting the uncertainty of the model output.

[0071] The output paradigm refers to a predefined, standardized output format and structure used to standardize the output of the visual language large model in the autonomous driving planning task. In the embodiment of the present application, it can be understood as a structured output template containing the reasoning process and the final parameters. Specifically, the output paradigm requires the model to not only give the final planner parameters, but also to show the complete reasoning process, including the analysis of vehicle interactions in the future, the lane change analysis based on lateral decisions, the analysis of longitudinal parameter generation, and the final parameter summary. The design of the output paradigm serves several important purposes: First, it ensures that the information output by the model is comprehensive and structured, which is convenient for subsequent system processing and execution. Secondly, by requiring the display of the reasoning process, it improves the interpretability and traceability of the model's decisions, which helps the safety verification and optimization of the system. Furthermore, this structured output helps to evaluate the decision quality of the model and provide a basis for the continuous improvement of the model. Finally, by standardizing the output format, it simplifies the interface between the model and other components of the autonomous driving system, and improves the integration efficiency and reliability of the entire system.

[0072] Specifically, the system first inputs the second sample thinking sub-chain into the visual language large model. As a high-level reasoning framework, the second sample thinking sub-chain is combined with the previously generated middle-level decisions to jointly guide the model to conduct deeper and more specific parameter reasoning. This combined approach has its unique advantages: the middle-level decision provides a macro-understanding and preliminary strategy for the current driving scenario, while the second sample thinking sub-chain provides a reasoning path for converting these macro-strategies into specific parameters. This combination ensures that the model does not deviate from the overall driving strategy when generating specific parameters, thereby improving the consistency and rationality of the decision.

[0073] Based on the second sample thinking subchain and the middle-level decision, the system further determines the output targets of the visual language large model. These output targets clearly specify the specific parameter types that the model needs to generate, including lane centerline tracking information required for lateral control, and speed limits, maximum acceleration, and maximum deceleration required for longitudinal control. By clearly defining the output targets, the system ensures that the output of the model directly corresponds to the needs of the autonomous driving control system, improving the practicality and executable nature of the output.

[0074] At the same time, the system also restricts the behavior of the visual language large model by setting input paradigms and output paradigms. The input paradigm specifies the standard format of scene information received by the model, including the vehicle status, surrounding vehicle information, and parameter selection range. This standardized input format ensures that the model can receive complete and accurate scene information, providing sufficient data support for making accurate decisions. The output paradigm defines the standard structure of the model output, requiring the model to not only give the final planning parameters, but also to show the complete reasoning process. This design greatly improves the interpretability and traceability of model decisions, and helps the safety verification and continuous optimization of the system.

[0075] Through this structured and constrained approach, the visual language model can eventually generate sample autonomous driving planning parameters for the current and next stages of the autonomous driving goal. These parameters include not only the control instructions that need to be executed immediately, but also the prediction and planning of recent driving behaviors. For example, for a lane change operation, the model will not only give the steering angle and acceleration that need to be executed at the moment, but also estimate the target lane position and speed after the lane change is completed.

[0076] S404: Obtain simulation results of sample autonomous driving planning parameters, and adjust the sample thinking chain according to the simulation results.

[0077] Specifically, the system first performs simulation tests on sample autonomous driving planning parameters generated by the visual language large model. This simulation test simulates the real road environment and traffic conditions, including various common and extreme situations, to comprehensively evaluate the feasibility and safety of the planning parameters. The simulation process not only takes into account the behavior of the vehicle itself, but also simulates the reactions of other traffic participants, as well as various possible environmental changes. This comprehensive simulation test is crucial for the safety verification of the autonomous driving system, as it can fully expose potential problems and risks without endangering actual road safety.

[0078] After the simulation test is completed, the system will conduct a detailed analysis of the simulation results. Based on the analysis of the simulation results, the system will make corresponding adjustments to the sample thinking chain. This adjustment is a delicate and complex process that may involve multiple levels: For example, if the simulation results show that the planning parameters perform poorly in certain specific types of scenarios, the system may adjust the scenario understanding and mid-level decision generation logic in the first sample thinking sub-chain to improve the model's recognition and processing capabilities for such scenarios. Secondly, if it is found that certain specific parameters (such as acceleration, steering angle, etc.) are improperly selected, the system may adjust the parameter generation logic in the second sample thinking sub-chain, such as modifying the parameter selection range or adjusting the priority of parameter generation. In addition, if the simulation results show that the model's performance in certain aspects (such as safety, efficiency, etc.) is not balanced enough, the system may adjust the weight distribution in the thinking chain to better balance the various performance indicators.

[0079] Based on the above embodiment, as an optional embodiment, the step of S404, obtaining simulation results of sample autonomous driving planning parameters, may further include the following steps: S601. Input the sample autonomous driving planning parameters into the simulator to obtain a trajectory prediction within a preset time period.

[0080] Specifically, the system first inputs the sample autonomous driving planning parameters generated by the visual language large model into a high-precision simulator. These parameters usually include lateral control information and longitudinal control information. As a complex software system, the simulator can simulate the physical characteristics of the real world, traffic rules and the behavior of other road users. It not only takes into account the dynamic characteristics of the autonomous driving vehicle, but also simulates factors such as tire-road friction, air resistance, and vehicle weight distribution to ensure the authenticity and reliability of the simulation results.

[0081] The simulator simulates the motion trajectory of the autonomous vehicle within a preset duration based on the input planning parameters. During the simulation, the system not only simulates the behavior of the autonomous vehicle, but also simulates the dynamic changes in the surrounding environment, including the movement of other vehicles, the activities of pedestrians, changes in traffic signals, etc. This comprehensive simulation ensures that the evaluation results can reflect the actual effects of the planning parameters in a complex and dynamic environment. The simulator records the detailed motion trajectory of the vehicle within a preset duration, including information such as position, speed, acceleration, direction, etc. at each time point, as well as the relative position and potential conflict points with other traffic participants.

[0082] Please refer to Figure 6 , Figure 6 This is a simulation schematic diagram provided by the present invention. Figure 6 The autonomous driving planning parameters based on the output of the large visual language model are shown. The intelligent driver model is used to perform a short-term simulation of the autonomous driving vehicle for 2 seconds, and the uniform speed model is used to make a 2-second trajectory prediction map of the moving obstacle.

[0083] S602: Score the predicted trajectory to obtain simulation results.

[0084] Specifically, the system can define a comprehensive and reasonable scoring standard. This set of standards usually includes multiple key dimensions to comprehensively evaluate the performance of autonomous driving. The main scoring dimensions may include: safety (such as the minimum safe distance from other vehicles, the number of emergency braking, etc.), efficiency (such as average speed, travel time, energy consumption, etc.), comfort (such as acceleration change rate, steering smoothness, etc.), rule compliance (such as response to traffic signals, lane keeping, etc.), and task completion (such as whether the destination is reached as expected). Each dimension will be given a corresponding weight to reflect its importance in the overall evaluation.

[0085] Next, the system will conduct a detailed analysis and scoring of the generated predicted trajectory based on this set of scoring criteria. This process usually involves complex data processing and calculations. For example, when evaluating safety, the system will calculate the minimum distance to other vehicles throughout the trajectory and count the number of potential collision risks; when evaluating efficiency, it will calculate the average speed and total travel time; when evaluating comfort, it will analyze the rate of change of acceleration and steering angle. These calculations not only take into account the static characteristics of the trajectory, but also analyze its dynamic changes to fully reflect the quality of autonomous driving decisions.

[0086] S405. Integrate the adjusted sample thinking chain into the visual language model.

[0087] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the autonomous driving planning method based on the visual language large model.

[0088] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the autonomous driving planning method based on the visual language large model provided by the above-mentioned methods.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the autonomous driving planning method based on the visual language large model provided by the above-mentioned methods.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An autonomous driving planning method based on a large visual language model, characterized in that: include: Obtain driving scene information; The driving scene information is input into the visual language big model, and the autonomous driving planning parameters output by the visual language big model are obtained, wherein the visual language big model is obtained by fine-tuning the sample driving scene information and the thinking chain, and the thinking chain is used to assist the visual language big model in performing multi-stage reasoning analysis on the sample driving scene information.

2. The autonomous driving planning method based on a visual language large model according to claim 1, characterized in that: The obtaining of driving scene information includes: Get driving scenes; Acquire road elements in the driving scene; Obtaining a bird's-eye view of the road elements; The bird's-eye view image is converted into a text description to obtain driving scene information.

3. The autonomous driving planning method based on a visual language large model according to claim 2, characterized in that: The road elements include lanes, dynamic objects, and static objects, and obtaining a bird's-eye view composed of the road elements includes: Converting the dynamic object into an arrow, wherein the arrow is used to represent the relative motion trend of the dynamic object; Converting the lane into a lane graph, wherein the lane graph includes nodes and edges, the nodes represent the center lines of the lanes, the edges include successor edges and adjacent edges, the successor edges represent nodes of the same lane, and the adjacent edges represent nodes of adjacent lanes; Construct a bird's-eye view corresponding to the arrow, the static object, and the lane map.

4. The autonomous driving planning method based on a visual language large model according to any one of claims 1 to 3, characterized in that: The fine-tuning process of the visual language large model includes: Obtain sample driving scene information; Determining a sample thinking chain based on the sample driving scenario information; Inputting the sample driving scene information and the sample thought chain into the visual language big model, and obtaining the sample autonomous driving planning parameters output by the visual language big model; Acquire simulation results of the sample autonomous driving planning parameters, and adjust the sample thinking chain according to the simulation results; Integrate the adjusted sample thought chain into the visual language model.

5. The autonomous driving planning method based on a visual language large model according to claim 4, characterized in that: The sample thinking chain includes a first sample thinking sub-chain and a second sample thinking sub-chain, and the first sample thinking sub-chain is prior to the second sample thinking sub-chain in logical order; The first sample thinking sub-chain is used to assist the visual language large model in making a mid-level decision on the sample driving scene, wherein the mid-level decision includes a lateral decision and a longitudinal decision, wherein the lateral decision is used to characterize a lane change operation, and the longitudinal decision is used to characterize a speed control operation; Among them, the second sample thinking sub-chain is used to assist the visual language large model in determining the autonomous driving planning parameters at different stages based on the middle-level decision.

6. The autonomous driving planning method based on a visual language large model according to claim 5, characterized in that: The step of inputting the sample driving scene information and the sample thought chain into the visual language big model and obtaining the sample autonomous driving planning parameters output by the visual language big model includes: Inputting the sample driving scene information and the first sample thinking sub-chain into the visual language big model to obtain the middle-level decision; The second sample thinking sub-chain is input into the visual language big model, and the output target of the visual language big model is determined through the second sample thinking sub-chain and the middle-level decision, as well as the input examples and output examples of the visual language big model are restricted, so as to obtain the sample autonomous driving planning parameters of the autonomous driving target in the current stage and the next stage.

7. The autonomous driving planning method based on a visual language large model according to claim 4, characterized in that: The obtaining of the simulation result of the sample autonomous driving planning parameter includes: Inputting the sample autonomous driving planning parameters into a simulator to obtain a trajectory prediction within a preset time period; The predicted trajectory is scored to obtain a simulation result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the autonomous driving planning method based on the visual language large model as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the autonomous driving planning method based on the visual language large model as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the autonomous driving planning method based on the visual language large model as described in any one of claims 1 to 7 is implemented.

Images