Automatic driving vehicle path planning method and device based on large language model, equipment and medium

Abstract

The invention relates to an automatic driving vehicle path planning method and device based on a large language model, equipment and a medium. The method comprises the steps that a traffic scene RGB image, a traffic scene depth image, laser radar point cloud data and inertial measurement unit data of an automatic driving vehicle are acquired; performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the laser radar point cloud data and the inertial measurement unit data to construct multi-modal traffic scene information; calling a preset large language model to determine the motion state of the autonomous vehicle and natural language description of traffic targets around the autonomous vehicle according to the multi-mode traffic scene information; and determining the attention weight of the target node by adopting an attention mechanism in a large language model according to the natural language description of the traffic target, and determining the optimal driving path by adopting a preset path planning algorithm according to the attention weight of the target node, the motion state of the autonomous vehicle and the target position. According to the invention, the adaptive capability of path planning is obviously improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a path planning method for an autonomous driving vehicle based on a large language model, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Path planning is one of the key steps in intelligent driving technology. In simple environments, path planning decisions based on information from a single sensor can achieve good results. However, in complex outdoor terrains, due to the lack of sufficient environmental information, the optimal path that can be achieved will be insufficiently robust and adaptable due to changes in environmental information. For example, purely visual path planning solutions will be affected by light and occlusion; and the perception of the environment by lidar will be affected by bad weather, causing noise in the propagation of the laser beam.

[0003] To sum up, the purely visual path planning scheme in the prior art will be affected by light and occlusion; and the perception of the environment by lidar will be affected by bad weather and the propagation of the laser beam will produce noise and other problems. The applicant has made corresponding explorations in order to solve this problem. Summary of the invention

[0004] The purpose of this application is to solve the above-mentioned problems and to provide an autonomous driving vehicle path planning method based on a large language model, a corresponding device, an electronic device and a computer-readable storage medium.

[0005] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0006] A path planning method for an autonomous driving vehicle based on a large language model is proposed to meet one of the purposes of this application, including:

[0007] In response to the instruction to plan the path of the autonomous driving vehicle, the autonomous driving vehicle obtains the traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data;

[0008] Performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, respectively, to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets, and their corresponding target categories;

[0009] Calling a preset large language model to determine a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it;

[0010] The attention mechanism in the large language model is used to determine the attention weight of the target node according to the natural language description of the traffic target, and the preset path planning algorithm is used to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position, so as to complete the path planning of the autonomous driving vehicle.

[0011] Optionally, the step of performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information includes:

[0012] A preset convolutional neural network is used to extract low-level feature information corresponding to each traffic target in the traffic scene RGB image, wherein the low-level feature information includes edges, textures, and contours of the traffic targets;

[0013] Calling a preset deep convolutional neural network to perform feature extraction on the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative positions between each traffic target, and using a preset feature extraction algorithm to extract local features of each traffic target in the traffic scene depth image;

[0014] A pre-trained semantic segmentation network is used to perform semantic segmentation on the lidar point cloud data, and the low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target and the local features of each traffic target are integrated to determine the different traffic targets in the traffic scene and their corresponding target categories.

[0015] Optionally, after the step of obtaining the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data of the autonomous driving vehicle, the following steps are included:

[0016] Using a Bayesian filter to update the pose estimation through probabilistic reasoning based on the traffic scene RGB image, the traffic scene depth image, the LiDAR point cloud data, and the inertial measurement unit data;

[0017] The local map is continuously updated based on the updated pose estimation. When sufficient pose information is obtained, the local map is expanded to a global map to construct a complete three-dimensional traffic scene representation.

[0018] Optionally, the step of calling a preset large language model to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information includes:

[0019] Calling a preset large language model, temporally aligning and spatially aligning and fusing the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, so as to generate a context embedding including the motion state of the autonomous driving vehicle and the traffic targets around it;

[0020] A natural language description of the motion state of the autonomous driving and the traffic targets around it is generated based on the fused context embedding.

[0021] Optionally, the step of using the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle, and the target position includes:

[0022] Generate a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it for path planning based on a preset large language model;

[0023] Constructing a three-dimensional traffic scene representation at a certain time step, assigning attention weights to target nodes based on the attention mechanism in the large language model according to the target mask generated by segmentation of the traffic scene RGB image and the natural language description of the traffic target, so as to determine the attention weight of the target node;

[0024] A preset path planning algorithm is used to perform the most feasible path planning exploration according to the attention weight of the target node. Every time new sensor data and large language model feedback are received, the three-dimensional traffic scene representation is updated, and the target position and driving path are recalculated to determine the optimal driving path of the autonomous vehicle.

[0025] Optionally, the traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles or buildings; the motion state includes current position, yaw angle, speed and acceleration.

[0026] Optionally, the large language model includes a Transformer model or a GPT model; the basic network architecture of the semantic segmentation network includes a PointNet++ network or a RangeNet++ network; the feature extraction algorithm includes a SIFT feature extraction algorithm or an ORB feature extraction algorithm; and the path planning algorithm includes an A* algorithm, a Dijkstra algorithm or an RRT algorithm.

[0027] A path planning device for an autonomous driving vehicle based on a large language model is provided to meet another purpose of the present application, comprising:

[0028] A data acquisition module, configured to respond to an instruction to perform path planning on the autonomous driving vehicle, and acquire a traffic scene RGB image, a traffic scene depth image, a lidar point cloud data, and an inertial measurement unit data of the autonomous driving vehicle;

[0029] a feature extraction module, configured to perform feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, respectively, to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets, and their corresponding target categories;

[0030] A natural language description module, configured to call a preset large language model to determine a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it;

[0031] The path planning module is configured to use the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position to complete the path planning of the autonomous driving vehicle.

[0032] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the autonomous driving vehicle path planning method based on a large language model described in the present application.

[0033] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the large language model-based autonomous driving vehicle path planning method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0034] Compared with the prior art, the present application aims at solving the problems that the purely visual path planning scheme in the prior art is affected by light and occlusion, and the laser radar perception of the environment is affected by bad weather and the propagation of the laser beam produces noise. The present application includes but is not limited to the following beneficial effects:

[0035] First, by integrating information from multiple sensors such as vision, LiDAR, and inertial measurement unit (IMU), the system can obtain a more comprehensive and accurate environmental perception. The complementarity of different sensors enables the system to maintain a high level of perception accuracy in complex or harsh environments. For example, visual sensors can provide detailed scene information, while LiDAR can accurately perceive the distance and shape of objects, and IMU provides dynamic motion information. Through the fusion of this information, the system has a more accurate understanding of the environment, thereby greatly reducing decision-making errors caused by the singleness of the sensor and improving the reliability and safety of the overall decision.

[0036] Secondly, this application uses a large language model to quickly analyze and process massive amounts of traffic environment data. Since path planning usually requires real-time response to changes in complex environments, combining a large language model can effectively improve the system's computing efficiency and response speed. The advantages of large language models in language understanding and reasoning enable the system to quickly extract useful information from a variety of sensors and quickly generate reasonable path planning decisions based on the current situation. This efficient data processing capability can ensure that the system responds promptly in a rapidly changing environment and avoids safety hazards caused by delays.

[0037] Third, the road planning system of this application has a high degree of adaptability and can dynamically adjust the path according to the real-time perceived environmental information. Especially in complex outdoor environments, factors such as obstacles, road conditions, and climate change always affect path selection. The large language model can not only combine multi-sensor information and perform real-time data analysis, but also optimize path selection through learning and reasoning to ensure that path planning can be adjusted in time under different environmental conditions to ensure driving safety and efficiency. For example, when encountering temporary obstacles or sudden weather changes, the system can recalculate a new path in the shortest time to avoid safety accidents.

[0038] Fourthly, this application uses multimodal information fusion technology to greatly enhance the robustness of path planning. Different types of sensors can complement each other. When one sensor fails or is interfered with, other sensors can continue to provide valid data, thereby ensuring the overall stability and reliability of the system. This robustness of multi-sensor fusion enables the system to provide efficient and accurate path planning decisions under complex environmental conditions (such as low light, bad weather, or dense obstacle environments), reducing safety risks caused by single sensor failure or interference.

[0039] Fifth, this application overcomes the limitations of traditional path planning algorithms. Traditional path planning methods often rely on a single sensor or rule engine and are easily limited by factors such as environmental complexity and sensor accuracy. This application overcomes the limitations of traditional methods by combining data from multiple sensors and the processing power of large language models. The system can not only handle dynamically changing complex environments, but also has strong generalization capabilities, can adapt to different scenarios and tasks, and provide more intelligent and flexible path planning.

[0040] Furthermore, through the combination of multimodal information fusion and large language model, this application provides an innovative path planning method, which significantly improves the perception, decision-making and adaptability of the path planning system in complex and dynamic environments, enhances the reliability, robustness and response speed of the system, overcomes the limitations of traditional technologies in complex environments, and provides stronger guarantees for the application of automation and intelligent systems under various harsh conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0042] Figure 1 An exemplary network architecture used by the autonomous driving vehicle path planning system based on a large language model in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a process of a path planning method for an autonomous driving vehicle based on a large language model in an embodiment of the present application;

[0044] Figure 3 This is an exemplary network architecture of the multi-sensor information fusion module in the embodiment of the present application;

[0045] Figure 4 This is a principle block diagram of a path planning device for an autonomous driving vehicle based on a large language model in an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0048] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0050] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0051] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0052] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0053] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0054] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0055] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0056] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0057] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0058] See also Figure 1 The large language model-based autonomous driving vehicle path planning method of the present application can be implemented based on the large language model-based autonomous driving vehicle path planning system, which is mainly composed of a data acquisition module, an information fusion module, a path planning module and a decision execution module.

[0059] In some embodiments, the data acquisition module integrates a variety of sensors, including visual sensors, laser radar (LiDAR), GPS modules and inertial measurement units (IMU), etc. Among them, the visual sensor uses the Realsense D455 depth camera to capture image information of the surrounding environment; the laser radar uses the Mid-360LiDAR to measure the distance to the surrounding objects and generate a high-precision environmental point cloud; the inertial measurement unit (IMU) is built into the laser radar, integrating a three-axis accelerometer and a gyroscope, providing the acceleration and angular velocity motion state information of the mobile platform to help calculate the posture change; the GPS module can obtain global real-time location information to ensure the global positioning accuracy of the path. Millimeter wave radar is used to detect dynamic obstacles in the environment, especially in severe weather conditions, to provide additional environmental perception capabilities. All sensor data is transmitted through a (CAN bus) high-bandwidth interface, and the data is transmitted to the central processing unit in real time to ensure the timeliness and accuracy of the information.

[0060] In some embodiments, the information fusion module uses multi-sensor data fusion technology to comprehensively process information from different sensors. This module uses fusion algorithms and filtering algorithms to ensure the accuracy and completeness of information, thereby compensating for the data loss and noise interference that may occur in a single sensor in a complex environment. The fusion algorithm analyzes the visual data, extracts features and fuses them with the lidar data to ensure the consistency of the information. The filtering algorithm introduces Kalman filters and particle filters to process the noise in the sensor data and improve the estimation accuracy of the position and posture. The path planning module is based on the path planning algorithm of the graph, and uses path planning algorithms such as the A* algorithm combined with the reasoning ability of the multimodal large language model to achieve the optimal path search. The algorithm can analyze historical data, adjust the plan according to real-time data, handle sudden obstacles or environmental changes, and ensure the safety of the path and the ability to dynamically replan. Among them, the multimodal large language model model can intelligently analyze, process and filter the collected data by training to understand the semantics of different sensor data, including processing the environmental information described in the text (such as map information, obstacle description), and incorporating this information into the path planning considerations. The decision execution module converts the path planning results into specific execution instructions to control the mobile platform to travel along the planned path. The system monitors environmental changes in real time and dynamically adjusts the driving path based on new data.

[0061] Based on the above exemplary scenarios, please refer to Figure 2 In one embodiment, the large language model-based autonomous driving vehicle path planning method of the present application includes:

[0062] Step S10, responding to an instruction to perform path planning on the autonomous driving vehicle, acquiring a traffic scene RGB image, a traffic scene depth image, a lidar point cloud data, and an inertial measurement unit data of the autonomous driving vehicle;

[0063] The autonomous driving vehicle path planning system based on the large language model in the autonomous driving vehicle can respond to the command of path planning for the autonomous driving vehicle, and obtain the traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data of the autonomous driving vehicle;

[0064] Specifically, see Figure 3 The exemplary structure of the multi-sensor information fusion module is as follows: Figure 2 As shown, it is necessary to integrate information from different data sources such as visual sensors, laser radars (LiDAR), inertial measurement units (IMU), etc. The core task of this module is to ensure the accuracy and timeliness of data fusion. Among them, the visual sensors include RGB-D cameras, etc., and the laser radars include Mid-360LiDAR, etc.

[0065] The RGB-D camera is used to obtain traffic scene RGB images and traffic scene depth images corresponding to different traffic scenes, wherein the preprocessing includes image calibration and screening, and the checkerboard method is used for camera calibration to reduce the error caused by lens distortion. The calibrated camera data can be used for depth information extraction of depth images and image blur analysis to ensure that the collected data is clear and of reliable quality.

[0066] The laser radar point cloud data can use Mid-360LiDAR to capture the surrounding 3D point cloud information, and use the voxel grid method to downsample the original laser radar point cloud data to reduce the data size and computational complexity. In addition, the unnecessary point cloud data is filtered out through the ROI frame to reduce the storage burden and processing time.

[0067] Inertial measurement unit data (IMU data) includes high-frequency linear acceleration and angular velocity, etc., which are used to extract the position information of key frames and perform data calibration. IMU data reduces noise and improves accuracy through Kalman filtering and complementary filtering algorithms. The preprocessed IMU data can be combined with the feature data of lidar and visual sensors to improve the overall perception accuracy.

[0068] like Figure 2 As shown, the information fusion module proposed in this embodiment includes a modal encoder, a visual encoder extracts visual features, and generates a feature map Where N is the number of spatial locations and d is the feature dimension; the point cloud encoder architecture uses the PointNet++ network to process the lidar point cloud data in the lidar to generate a feature vector that captures the spatial structure.

[0069] Furthermore, the size of the traffic scene RGB image is adjusted to 224×224 pixels, and the mean and standard deviation based on the ImageNet dataset are normalized. Random cropping, horizontal flipping, rotation and other techniques are used to enhance the data, increase robustness and prevent model overfitting; the lidar point cloud data is voxelized to reduce data redundancy, and the k-nearest neighbor statistical filtering statistical outlier removal method is used to eliminate outliers and noise. The traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data (IMU data) are aligned based on the timestamp to ensure the time consistency between the information of each sensor. The synchronization technology uses a global clock or software clock to ensure that the information fusion module receives data at the same time.

[0070] More specifically, the multi-sensor information fusion module is constructed. First, the information from different data sources such as the inertial measurement unit (IMU) is received, and the data from the depth camera and lidar are preprocessed and preliminarily screened to extract real-time LiDAR (Light Detection and Ranging) and visual features, and the extracted feature information is stored in the local map. Subsequently, the raw data of the IMU is preprocessed to extract the pose information of the key frames and high-frequency motion data. Finally, by scanning the context, the IMU matches and supplements the motion trajectory of the fusion information of the radar and camera to avoid pose drift caused by feature matching errors, so as to maximize the robustness.

[0071] Among them, the RGB-D camera mounted on the robot is used to shoot videos of different scenes, and the RGB images and depth images are collected through the original video frames; the camera sensor needs to be calibrated before use, because the camera lens will distort the obtained image to a certain extent. Here, a checkerboard black and white picture is used to compare the camera output image with the original image to calculate the calibration error. The images and depth images collected in different environments are judged as blurred images, and clear images are selected. These images are preprocessed and saved to obtain the initial image data;

[0072] Among them, Mid-360LiDAR is used to capture the field of view (FoV) information around the robot, and the speed of light and the time of flight (ToF) between the transmitted and received lasers are calculated through the built-in timer and timer to obtain the position status information of the surrounding traffic targets. Since the original point cloud of Mid-360LiDAR has a high resolution, covers a wide distance, and has a high computational cost, downsampling (Downsampling) and the use of the region of interest (ROI) box to filter irrelevant point cloud information are used through the voxel grid method (Voxel Grid Filter) to reduce the computational burden and storage requirements. The remaining position status information is saved in the processor in the format of point cloud information; Among them, the inertial measurement unit (IMU) is used to obtain the dynamic information of traffic targets or equipment for further filtering and calibration, which can reduce the posture error of the key frame and improve the accuracy of visual closed-loop detection. In addition, IMU data can be used to solve the problems of point cloud distortion and rotation drift caused by LiDAR or visual movement; and, if there are slight errors in the feature matching results of the camera or LiDAR, such as feature matching failure due to dynamic environment or partial occlusion, IMU data can provide continuous dynamic constraints to avoid error accumulation.

[0073] The data collection process needs to take into account changes in ambient lighting, irregular terrain, dynamic obstacles, and weather changes to ensure the diversity and richness of the data set. The visual image size is adjusted and normalized to 224*224 pixels and normalized using the mean and standard deviation of the imageNet data set. Data enhancement is performed through random cropping, flipping, rotation, and other techniques. Color jitter is used to increase data diversity and prevent overfitting; the lidar point cloud data uses point cloud voxelization to reduce data size and computational complexity, and statistical outlier removal is performed to eliminate stray points. Sensor data is synchronized based on timestamps to ensure temporal consistency between modalities, and the calibration parameters provided in the data set are used to align spatial data from different sensors. Text data is tokenized using a subword tagger (WordPiece), and then pre-trained embeddings of models such as the Transformer model and the GPT model are used to represent the text data, where the GPT model includes the GPT-4 model and the like.

[0074] To design a path planning algorithm in an unknown environment, it is necessary to construct a semantic 3D traffic scene representation of the environment through an autonomous vehicle and generate a plan based on this scene using a large language model. The goal of the autonomous vehicle is to find a target location in the environment and create a rich scene representation around the traffic target for downstream planning. The reasoning ability of the large language model (LLM model) is used to understand the semantic priority of the environment to guide the robot to reach the target area. The autonomous vehicle takes RGB images, depth information, and pose as input at each timestamp, and identifies objects in the RGB image through the segmentation module and generates masks for them. At this time, the large language model (LLM model) acts as a prune to identify the most important objects in the scene based on the semantic understanding of the environment, and detects segments and prunes unnecessary segments. According to the similarity criterion, these detected segments are converted into a 3D traffic scene representation to provide semantic information for the path planning algorithm based on the large language model (LLM model), where each target is represented by a node. The target node stores information about the location of the object, the point cloud, the frame where the target is detected, the mask and label of the target in the frame, and the semantic description of the target in the scene. The autonomous vehicle then selects a node for nearest exploration based on the generated modal weights to find the target area. The large language model is combined with path planning and uses a modular representation of the 3D traffic scene to decide whether to continue exploring the environment or approach the target already detected in the scene. During this process, the autonomous vehicle stores the relevant information for each frame in the memory module. If the autonomous vehicle determines that there is no good opportunity to approach the target area in the 3D traffic scene, it will continue to explore the scene and build a 3D traffic scene representation until it reaches the target.

[0075] The autonomous driving vehicle constructs a modular 3D traffic scene, segments and annotates the RGB image frame of the traffic scene at the current time step t, adds target labels for record storage, and then converts the target annotation set into text prompts and corresponding masks. At the same time, the point cloud information and posture information at the current time step t are combined to generate a 3D object node.

[0076] Among them, in order to enable the large language model (LLM model) to effectively process and reason about multimodal data, the multi-sensor information is converted into a text description suitable for the large language model (LLM model) input. Multimodal information requires data preprocessing before conversion, including data source cleaning and enhancement methods. Based on factors such as performance, compatibility, and computing requirements, the GPT-4 model is selected as the large language model (LLM model) architecture to ensure that the embedding generated by the large language model (LLM model) is compatible with the embedding generated by the sensor encoder. An API is created to complete the communication between the large language model (LLM model) and the fusion module, including data exchange protocols, handling data serialization / deserialization, and ensuring that the output of the large language model (LLM model) is correctly interpreted by the fusion module. Then, the feedback signal of the large language model (LLM model) is integrated into the parameter adjustment mechanism of the fusion module, which is conducive to refining the weight module and updating the fusion strategy to improve system performance.

[0077] Among them, the modal weight mechanism in the decision model based on the large language model (LLM model) generates the attention weight of the target node based on the environment, including attention fusion and cross-modal conversion. According to the reasoning process of the large language model (LLM model), different prompt structures, information input and context clues, the response of the large language model (LLM model) to the weighting of modalities and strategy selection is optimized. At the same time, ablation experiments are carried out by disabling or modifying the information of some sensors in the decision module based on the large language model (LLM model) to evaluate the impact on the overall performance when some modal information is lost. And a sensitivity analysis is carried out by changing the learning rate and attention mechanism of the large language model (LLM model) to determine the sensitivity of the large language model (LLM model) to parameters to ensure stability and efficiency.

[0078] Among them, the path planning module uses goal guidance and environmental mapping to construct the obstacle map, boundary map and target map of the current scene. In the scene initialization state, the target map and boundary map are the same; when the multimodal information is processed to construct the current scene, the target map is constructed according to the robot approaching the target area selected by the large language model (LLM model), and then the target map and obstacle map are used as the basis for the selection of the path planning algorithm.

[0079] In some embodiments, after the step of acquiring the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data of the autonomous driving vehicle, the following steps are included:

[0080] Step S101, using a Bayesian filter to update the pose estimation through probabilistic reasoning based on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data;

[0081] Step S102: continuously update the local map according to the updated pose estimation, and when sufficient pose information is obtained, expand the local map into a global map to construct a complete three-dimensional traffic scene representation.

[0082] Step S20, respectively extracting features from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets, and their corresponding target categories;

[0083] After acquiring the traffic scene RGB image, traffic scene depth image, laser radar point cloud data and inertial measurement unit data of the autonomous driving vehicle, feature extraction is performed on the traffic scene RGB image, the traffic scene depth image, the laser radar point cloud data and the inertial measurement unit data respectively to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets and their corresponding target categories;

[0084] In some embodiments, the step of performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information includes:

[0085] Step S201, using a preset convolutional neural network to extract low-level feature information corresponding to each traffic target in the traffic scene RGB image, wherein the low-level feature information includes the edge, texture, and contour of the traffic target;

[0086] Step S202: calling a preset deep convolutional neural network to perform feature extraction on the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative positions between each traffic target, and using a preset feature extraction algorithm to extract local features of each traffic target in the traffic scene depth image;

[0087] Step S203: Use a pre-trained semantic segmentation network to perform semantic segmentation on the lidar point cloud data, and fuse the low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target, and the local features of each traffic target to determine the different traffic targets in the traffic scene and their corresponding target categories.

[0088] Specifically, after the preprocessing stage is completed, features are extracted from the traffic scene RGB image, traffic scene depth image and lidar point cloud data and input into the feature extraction module, and a three-dimensional traffic scene representation is constructed. Convolutional neural network (CNN) and deep learning technology can be used to extract features from RGB images, and deep images can be supplemented with feature extraction through deep learning networks, such as hybrid feature extraction using deep convolutional network (DCNN) and feature extraction algorithms. The lidar provides point cloud data for the observation model, and the point cloud features use ground segmentation and semantic segmentation networks to distinguish different traffic targets in the point cloud and their corresponding target categories; high-frequency angular velocity and acceleration information in IMU data can be extracted for posture estimation and motion prediction, and the predicted pose of the Bayesian filter can be corrected through the system fusion model. The composition algorithm uses the corrected predicted pose to create a local map, and when the best pose is obtained, the local map is updated to the global map. The global map is Gaussian blurred to construct a new observation model, and the three-dimensional traffic scene forms a complete closed loop.

[0089] In some embodiments, the basic network architecture of the semantic segmentation network includes a PointNet++ network or a RangeNet++ network, etc.; the feature extraction algorithm includes a SIFT feature extraction algorithm or an ORB feature extraction algorithm, etc.

[0090] Step S30: calling a preset large language model to determine a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it;

[0091] After respectively performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information, a preset large language model is called to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the position information of the traffic targets around it; wherein the traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles or buildings, etc.; the motion state includes the current position, yaw angle, speed and acceleration, etc., and the large language model includes a Transformer model or a GPT model, etc.

[0092] In some embodiments, the step of calling a preset large language model to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information includes:

[0093] Step S301, calling a preset large language model, temporally aligning and spatially aligning and fusing the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, so as to generate a context embedding including the motion state of the autonomous driving vehicle and the traffic targets around it;

[0094] Step S302: Generate a natural language description of the motion state of the autonomous driving and the traffic targets around it based on the fused context embedding.

[0095] Specifically, the multimodal traffic scene information is passed to the large language model, which needs to be converted into a description suitable for text input, and a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it. The natural language description of the traffic targets around the autonomous driving vehicle can be expressed as "there is a car 1 meter ahead and an obstacle 2 meters to the left", etc., which can be used as a natural language description of the traffic targets around the autonomous driving vehicle; the natural language description of the motion state of the autonomous driving vehicle can be expressed as "the robot moves forward at a speed of 1 meter per second", etc., which can be used as a natural language description of the motion state of the autonomous driving vehicle.

[0096] Data encoding uses a subword tagger (WordPiece) to segment text and generate semantic embeddings, which are aligned and fused with other sensor data. The text encoder encodes the text input based on the transformer model. The fusion module uses the large language model LLM to integrate the features of different modalities and connect the modal features. Where M is the number of modalities; the Transformer layer uses the attention mechanism and feedforward network to simulate complex dependencies and uniformly generate express.

[0097] Step S40: using the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position, so as to complete the path planning of the autonomous driving vehicle.

[0098] After calling the preset large language model to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information, the attention mechanism in the large language model is used to determine the attention weight of the target node according to the natural language description of the traffic target, and the preset path planning algorithm is used to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position to complete the path planning of the autonomous driving vehicle, wherein the path planning algorithm includes an A* algorithm, a Dijkstra algorithm or an RRT algorithm, etc.

[0099] In some embodiments, the steps of using the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle, and the target position include:

[0100] Step S401: Generate a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it based on a preset large language model to perform path planning;

[0101] Step S402, constructing a three-dimensional traffic scene representation at a certain time step, assigning attention weights to target nodes based on the target mask generated by segmenting the traffic scene RGB image and the natural language description of the traffic target according to the attention mechanism in the large language model, so as to determine the attention weight of the target node;

[0102] Step S403: Use a preset path planning algorithm to perform the most feasible path planning exploration according to the attention weight of the target node. Whenever new sensor data and large language model feedback are received, the three-dimensional traffic scene representation is updated, and the target position and driving path are recalculated to determine the optimal driving path of the autonomous vehicle.

[0103] Specifically, the GPT-4 model can be selected as the architecture of the large language model (LLM model), which does not constitute a limitation on this application. Data communication and interaction can be achieved through an application programming interface (API), and the API is designed. The API interface design takes into account serialization and deserialization to ensure smooth data transmission between modules. The API supports data transmission protocols such as JSON or Protobuf. Model optimization selects the best learning rate and attention mechanism through experiments, and adjusts parameters to improve the responsiveness to multimodal input. Ablation experiments are used to evaluate system performance when some data is missing, and to analyze the sensitivity of the large language model (LLM model) to different modal weights.

[0104] Path planning is performed based on the multimodal traffic scene semantic representation generated by the large language model (LLM model), and a three-dimensional traffic scene representation is constructed at a certain time step. Attention weights are assigned to target nodes based on the attention mechanism in the large language model according to the target mask generated by segmentation of the traffic scene RGB image and the natural language description of the traffic target to determine the attention weight of the target node. A preset path planning algorithm is used to perform the most feasible path planning exploration according to the attention weight of the target node. Whenever new sensor data and large language model feedback are received, the three-dimensional traffic scene representation is updated, and the target position and driving path are recalculated to determine the optimal driving path for the autonomous vehicle.

[0105] In a further embodiment, the decision results of the large language model (LLM model) are integrated into the path planning module, and the weight adjustment optimizes the weight parameters of the traffic scene RGB image, traffic scene depth image, lidar point cloud data, and the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal input to achieve the weight distribution of cross-modal information. The strategy selection is based on the pruning algorithm of the large language model (LLM model), which identifies key objects based on environmental information and ignores irrelevant parts. The sensitivity score evaluates the stability and efficiency of the model under different configurations by adjusting the learning rate and attention mechanism. .

[0106] In the path planning module, obstacle maps, boundary maps, and target maps are constructed for path calculation. In the initial state, the boundary map and the target map are the same, and the target area is updated as the exploration proceeds. Path calculation uses heuristic algorithms (such as A algorithm, Dijkstra algorithm) or optimization algorithms (such as RRT algorithm, D-Lite algorithm) to generate paths, combined with the semantic understanding and scenario analysis of the LLM model to determine the optimal driving path. Furthermore, during the execution of real-time path planning, changes in the traffic environment are monitored and the path is recalculated to avoid dynamic obstacles.

[0107] In some embodiments, the path planning method for an autonomous driving vehicle based on a large language model of the present application is represented by the following pseudo code:

[0108]

[0109] It can be seen from the above embodiments that, compared with the prior art, the present application aims at the problems that the purely visual path planning scheme in the prior art will be interfered by light and occlusion; and the perception of the environment by laser radar will be affected by bad weather and the propagation of the laser beam will produce noise. The present application includes but is not limited to the following beneficial effects:

[0110] First, by integrating information from multiple sensors such as vision, LiDAR, and inertial measurement unit (IMU), the system can obtain a more comprehensive and accurate environmental perception. The complementarity of different sensors enables the system to maintain a high level of perception accuracy in complex or harsh environments. For example, visual sensors can provide detailed scene information, while LiDAR can accurately perceive the distance and shape of objects, and IMU provides dynamic motion information. Through the fusion of this information, the system has a more accurate understanding of the environment, thereby greatly reducing decision-making errors caused by the singleness of the sensor and improving the reliability and safety of the overall decision.

[0111] Secondly, this application uses a large language model to quickly analyze and process massive amounts of traffic environment data. Since path planning usually requires real-time response to changes in complex environments, combining a large language model can effectively improve the system's computing efficiency and response speed. The advantages of large language models in language understanding and reasoning enable the system to quickly extract useful information from a variety of sensors and quickly generate reasonable path planning decisions based on the current situation. This efficient data processing capability can ensure that the system responds promptly in a rapidly changing environment and avoids safety hazards caused by delays.

[0112] Third, the road planning system of this application has a high degree of adaptability and can dynamically adjust the path according to the real-time perceived environmental information. Especially in complex outdoor environments, factors such as obstacles, road conditions, and climate change always affect path selection. The large language model can not only combine multi-sensor information and perform real-time data analysis, but also optimize path selection through learning and reasoning to ensure that path planning can be adjusted in time under different environmental conditions to ensure driving safety and efficiency. For example, when encountering temporary obstacles or sudden weather changes, the system can recalculate a new path in the shortest time to avoid safety accidents.

[0113] Fourthly, this application uses multimodal information fusion technology to greatly enhance the robustness of path planning. Different types of sensors can complement each other. When one sensor fails or is interfered with, other sensors can continue to provide valid data, thereby ensuring the overall stability and reliability of the system. This robustness of multi-sensor fusion enables the system to provide efficient and accurate path planning decisions under complex environmental conditions (such as low light, bad weather, or dense obstacle environments), reducing safety risks caused by single sensor failure or interference.

[0114] Fifth, this application overcomes the limitations of traditional path planning algorithms. Traditional path planning methods often rely on a single sensor or rule engine and are easily limited by factors such as environmental complexity and sensor accuracy. This application overcomes the limitations of traditional methods by combining data from multiple sensors and the processing power of large language models. The system can not only handle dynamically changing complex environments, but also has strong generalization capabilities, can adapt to different scenarios and tasks, and provide more intelligent and flexible path planning.

[0115] Furthermore, through the combination of multimodal information fusion and large language model, this application provides an innovative path planning method, which significantly improves the perception, decision-making and adaptability of the path planning system in complex and dynamic environments, enhances the reliability, robustness and response speed of the system, overcomes the limitations of traditional technologies in complex environments, and provides stronger guarantees for the application of automation and intelligent systems under various harsh conditions.

[0116] See also Figure 4 , a path planning device for an autonomous driving vehicle based on a large language model is provided to meet one of the purposes of the present application, including a data acquisition module 1100, a feature extraction module 1200, a natural language description module 1300 and a path planning module 1400. Among them, the data acquisition module 1100 is configured to respond to the instruction of path planning for the autonomous driving vehicle, and obtain the traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data of the autonomous driving vehicle; the feature extraction module 1200 is configured to extract features from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data and the inertial measurement unit data respectively to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, multiple different traffic targets and their corresponding target categories; the natural language description module 1300 is configured to The method is configured to call a preset large language model to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it; the path planning module 1400 is configured to use the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position, so as to complete the path planning of the autonomous driving vehicle.

[0117] Based on any embodiment of this application, please refer to Figure 5Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 5 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for autonomous driving vehicle path planning based on a large language model. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for autonomous driving vehicle path planning based on a large language model of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In this embodiment, the processor is used to execute Figure 4 The memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules in the large language model-based autonomous driving vehicle path planning device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0119] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the autonomous driving vehicle path planning method based on a large language model described in any embodiment of the present application.

[0120] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the autonomous driving vehicle path planning method based on a large language model described in any embodiment of the present application.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0122] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0123] In summary, through the combination of multimodal information fusion and large language model, this application provides an innovative path planning method, which significantly improves the perception, decision-making and adaptability of the path planning system in complex and dynamic environments, enhances the reliability, robustness and response speed of the system, overcomes the limitations of traditional technologies in complex environments, and provides stronger guarantees for the application of automation and intelligent systems under various harsh conditions.

Claims

1. A path planning method for an autonomous driving vehicle based on a large language model, characterized in that: include: In response to the instruction to plan the path of the autonomous driving vehicle, the autonomous driving vehicle obtains the traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data; Performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, respectively, to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets, and their corresponding target categories; Calling a preset large language model to determine a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it; The attention mechanism in the large language model is used to determine the attention weight of the target node according to the natural language description of the traffic target, and the preset path planning algorithm is used to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position, so as to complete the path planning of the autonomous driving vehicle.

2. The method for path planning of an autonomous driving vehicle based on a large language model according to claim 1, characterized in that: The steps of respectively extracting features from the traffic scene RGB image, the traffic scene depth image, the laser radar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information include: A preset convolutional neural network is used to extract low-level feature information corresponding to each traffic target in the traffic scene RGB image, wherein the low-level feature information includes edges, textures, and contours of the traffic targets; Calling a preset deep convolutional neural network to perform feature extraction on the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative positions between each traffic target, and using a preset feature extraction algorithm to extract local features of each traffic target in the traffic scene depth image; A pre-trained semantic segmentation network is used to perform semantic segmentation on the lidar point cloud data, and the low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target and the local features of each traffic target are integrated to determine the different traffic targets in the traffic scene and their corresponding target categories.

3. The method for path planning of an autonomous driving vehicle based on a large language model according to claim 1, characterized in that: After the steps of obtaining the traffic scene RGB image, traffic scene depth image, lidar point cloud data and inertial measurement unit data of the autonomous driving vehicle, the following steps are included: Using a Bayesian filter to update the pose estimation through probabilistic reasoning based on the traffic scene RGB image, the traffic scene depth image, the LiDAR point cloud data, and the inertial measurement unit data; The local map is continuously updated based on the updated pose estimation. When sufficient pose information is obtained, the local map is expanded to a global map to construct a complete three-dimensional traffic scene representation.

4. The method for path planning of an autonomous driving vehicle based on a large language model according to claim 1, characterized in that: The step of calling a preset large language model to determine the motion state of the autonomous driving vehicle and the natural language description of the traffic targets around it according to the multimodal traffic scene information includes: Calling a preset large language model, temporally aligning and spatially aligning and fusing the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, so as to generate a context embedding including the motion state of the autonomous driving vehicle and the traffic targets around it; A natural language description of the motion state of the autonomous driving and the traffic targets around it is generated based on the fused context embedding.

5. The method for path planning of an autonomous driving vehicle based on a large language model according to claim 3, characterized in that: The steps of using the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position include: Generate a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it for path planning based on a preset large language model; Constructing a three-dimensional traffic scene representation at a certain time step, assigning attention weights to target nodes based on the attention mechanism in the large language model according to the target mask generated by segmentation of the traffic scene RGB image and the natural language description of the traffic target, so as to determine the attention weight of the target node; A preset path planning algorithm is used to perform the most feasible path planning exploration according to the attention weight of the target node. Every time new sensor data and large language model feedback are received, the three-dimensional traffic scene representation is updated, and the target position and driving path are recalculated to determine the optimal driving path for the autonomous vehicle.

6. The method for path planning of an autonomous driving vehicle based on a large language model according to any one of claims 1 to 5, characterized in that: The traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles or buildings; the motion state includes current position, yaw angle, speed and acceleration.

7. The method for path planning of an autonomous driving vehicle based on a large language model according to any one of claims 1 to 5, characterized in that: The large language model includes a Transformer model or a GPT model; the basic network architecture of the semantic segmentation network includes a PointNet++ network or a RangeNet++ network; the feature extraction algorithm includes a SIFT feature extraction algorithm or an ORB feature extraction algorithm; the path planning algorithm includes an A* algorithm, a Dijkstra algorithm or an RRT algorithm.

8. A path planning device for an autonomous driving vehicle based on a large language model, characterized in that: include: A data acquisition module, configured to respond to an instruction to perform path planning on the autonomous driving vehicle, and acquire a traffic scene RGB image, a traffic scene depth image, a lidar point cloud data, and an inertial measurement unit data of the autonomous driving vehicle; a feature extraction module, configured to perform feature extraction on the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data, respectively, to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed by the motion state of the autonomous driving vehicle, a plurality of different traffic targets, and their corresponding target categories; A natural language description module, configured to call a preset large language model to determine a natural language description of the motion state of the autonomous driving vehicle and the traffic targets around it according to the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous driving vehicle and the location information of the traffic targets around it; The path planning module is configured to use the attention mechanism in the large language model to determine the attention weight of the target node according to the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path according to the attention weight of the target node, the motion state of the autonomous driving vehicle and the target position to complete the path planning of the autonomous driving vehicle.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images