Automatic driving vehicle path planning method and device based on large language model, equipment and medium

Abstract

The application relates to an automatic driving vehicle path planning method and device based on a large language model, equipment and a medium. The method comprises the following steps: acquiring a traffic scene RGB image, a traffic scene depth image, laser radar point cloud data and inertial measurement unit data of an automatic driving vehicle; performing feature extraction on the traffic scene RGB image, the traffic scene depth image, the laser radar point cloud data and the inertial measurement unit data respectively to construct multi-modal traffic scene information; calling a preset large language model to determine a motion state of the automatic driving vehicle and a natural language description of a traffic target around the automatic driving vehicle according to the multi-modal traffic scene information; adopting an attention mechanism in the large language model to determine an attention weight of a target node according to the natural language description of the traffic target; and adopting a preset path planning algorithm to determine an optimal driving path according to the attention weight of the target node, the motion state of the automatic driving vehicle and a target position. The application significantly improves the adaptive capability of path planning.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to an autonomous vehicle path planning method based on a large language model, a corresponding device, electronic equipment, and a computer-readable storage medium. Background Technology

[0002] Path planning is a crucial step in intelligent driving technology. In simple environments, path planning decisions based on information from a single sensor can achieve good results. However, in complex outdoor terrain, due to a lack of sufficient environmental information, the achievable optimal path may suffer from insufficient robustness and adaptability as environmental information changes. For example, purely visual path planning schemes are affected by lighting and occlusion; while relying on LiDAR for environmental perception is susceptible to noise caused by adverse weather conditions and the propagation of the laser beam.

[0003] In summary, existing vision-based path planning schemes are susceptible to interference from lighting and occlusion; while relying on lidar for environmental perception is affected by adverse weather conditions and noise generated by the propagation of the laser beam. The applicant has made corresponding explorations to address these issues. Summary of the Invention

[0004] The purpose of this application is to solve the above-mentioned problems by providing an autonomous vehicle path planning method, corresponding device, electronic device and computer-readable storage medium based on a large language model.

[0005] To achieve the various objectives of this application, the following technical solution is adopted:

[0006] A method for autonomous vehicle path planning based on a large language model, proposed to meet one of the purposes of this application, includes:

[0007] In response to instructions for path planning of autonomous vehicles, the system acquires RGB images of traffic scenes, depth images of traffic scenes, LiDAR point cloud data, and inertial measurement unit data of autonomous vehicles.

[0008] Feature extraction is performed on the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the lidar, and the data of the inertial measurement unit to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories.

[0009] A preset large language model is invoked to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it.

[0010] The attention mechanism in the large language model is used to determine the attention weight of the target node based on the natural language description of the traffic target. A preset path planning algorithm is then used to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle.

[0011] Optionally, the step of extracting features from the RGB image of the traffic scene, the depth image of the traffic scene, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information includes:

[0012] A pre-defined convolutional neural network is used to extract low-level feature information corresponding to each traffic target in the RGB image of the traffic scene. The low-level feature information includes the edges, textures, and contours of the traffic targets.

[0013] A preset deep convolutional neural network is invoked to extract features from the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative positions between each traffic target. A preset feature extraction algorithm is used to extract the local features of each traffic target in the traffic scene depth image.

[0014] A pre-trained semantic segmentation network is used to perform semantic segmentation on the lidar point cloud data. The low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target, and the local features of each traffic target are fused to determine the different traffic targets in the traffic scene and their corresponding target categories.

[0015] Optionally, after acquiring the RGB image of the traffic scene, the depth image of the traffic scene, the LiDAR point cloud data, and the inertial measurement unit data of the autonomous vehicle, the process includes:

[0016] A Bayesian filter is used to update the pose estimation based on the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the LiDAR, and the data of the inertial measurement unit through probabilistic reasoning.

[0017] The local map is continuously updated based on the updated pose estimation. When sufficient pose information is obtained, the local map is expanded into a global map to construct a complete 3D traffic scene representation.

[0018] Optionally, the step of calling a preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the surrounding traffic targets based on the multimodal traffic scene information includes:

[0019] The preset large language model is invoked to perform time and space alignment and fusion of the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data and the inertial measurement unit data, so as to generate a context embedding that includes the motion state of the autonomous vehicle and the traffic targets around it.

[0020] A natural language description of the motion state of the autonomous vehicle and its surrounding traffic targets is generated based on the fused context embedding.

[0021] Optionally, the steps of using the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, include:

[0022] Path planning is performed by generating natural language descriptions of the motion state of the autonomous vehicle and the surrounding traffic targets based on a preset large language model.

[0023] A 3D traffic scene representation is constructed at a certain time step. Based on the attention mechanism in the large language model, attention weights are assigned to the target nodes according to the target mask generated by segmenting the RGB image of the traffic scene and the natural language description of the traffic target, so as to determine the attention weight of the target node.

[0024] A preset path planning algorithm is used to explore the most feasible path planning based on the attention weight of the target node. Whenever new sensor data and large language model feedback are received, the 3D traffic scene representation is updated and the target position and driving path are recalculated to determine the optimal driving path of the autonomous vehicle.

[0025] Optionally, the traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles, or buildings; the motion state includes current position, yaw angle, speed, and acceleration.

[0026] Optionally, the large language model includes a Transformer model or a GPT model; the basic network architecture of the semantic segmentation network includes a PointNet++ network or a RangeNet++ network; the feature extraction algorithm includes a SIFT feature extraction algorithm or an ORB feature extraction algorithm; and the path planning algorithm includes an A* algorithm, a Dijkstra algorithm, or a RRT algorithm.

[0027] A path planning device for autonomous vehicles based on a large language model, provided for another purpose of this application, includes:

[0028] The data acquisition module is configured to respond to instructions for path planning of autonomous vehicles by acquiring RGB images of traffic scenes, depth images of traffic scenes, LiDAR point cloud data, and inertial measurement unit data of autonomous vehicles.

[0029] The feature extraction module is configured to extract features from the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the lidar, and the data of the inertial measurement unit, respectively, to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories.

[0030] The natural language description module is configured to call a preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it.

[0031] The path planning module is configured to use the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle.

[0032] An electronic device provided for another purpose of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the autonomous vehicle path planning method based on a large language model described in this application.

[0033] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the aforementioned autonomous vehicle path planning method based on a large language model, which, when invoked by a computer, executes the steps included in the corresponding method.

[0034] Compared to existing technologies, this application addresses the problems of pure vision-based path planning being affected by lighting and occlusion, and the environmental perception relying on lidar being affected by adverse weather conditions and noise generated by laser beam propagation. This application provides, but is not limited to, the following beneficial effects:

[0035] Firstly, this application integrates information from multiple sensors, including vision, LiDAR, and inertial measurement unit (IMU), enabling the system to achieve more comprehensive and accurate environmental perception. The complementarity of different sensors allows the system to maintain high perception accuracy even in complex or harsh environments. For example, vision sensors can provide rich scene information, while LiDAR can accurately perceive the distance and shape of objects, and the IMU provides dynamic motion information. Through the fusion of this information, the system's understanding of the environment becomes more precise, thereby significantly reducing decision-making errors caused by the reliance on a single sensor and improving the reliability and security of overall decision-making.

[0036] Secondly, this application employs a large language model, which enables rapid analysis and processing of massive amounts of traffic environment data. Since route planning typically requires real-time responses to complex environmental changes, combining it with a large language model can effectively improve the system's computational efficiency and response speed. The advantages of large language models in language understanding and reasoning allow the system to quickly extract useful information from multiple sensors and rapidly generate reasonable route planning decisions based on the current context. This efficient data processing capability ensures that the system reacts promptly in rapidly changing environments, avoiding safety hazards caused by delays.

[0037] Thirdly, the route planning system of this application possesses a high degree of adaptability, capable of dynamically adjusting the route based on real-time perceived environmental information. Especially in complex outdoor environments, factors such as obstacles, road conditions, and climate change constantly influence route selection. The large language model not only combines multi-sensor information for real-time data analysis but also optimizes route selection through learning and reasoning, ensuring that route planning can be adjusted promptly under different environmental conditions to guarantee driving safety and efficiency. For example, when encountering temporary obstacles or sudden weather changes, the system can recalculate a new route in the shortest possible time, preventing accidents.

[0038] Fourth, this application significantly enhances the robustness of path planning through multimodal information fusion technology. Different types of sensors can complement each other; if one sensor fails or is interfered with, the others can continue to provide valid data, thereby ensuring the overall stability and reliability of the system. This robustness of multi-sensor fusion enables the system to provide efficient and accurate path planning decisions even under complex environmental conditions (such as low light, severe weather, or dense obstacle environments), reducing safety risks caused by single sensor failure or interference.

[0039] Fifth, this application overcomes the limitations of traditional path planning algorithms. Traditional path planning methods often rely on a single sensor or rule engine, which is easily limited by factors such as environmental complexity and sensor accuracy. This application, however, overcomes these limitations by combining data from multiple sensors and the processing power of a large language model. The system can not only handle dynamically changing and complex environments but also possesses strong generalization capabilities, adapting to different scenarios and tasks and providing more intelligent and flexible path planning.

[0040] Furthermore, by combining multimodal information fusion with a large language model, this application provides an innovative path planning method that significantly improves the perception, decision-making, and adaptive capabilities of path planning systems in complex and dynamic environments, enhances the system's reliability, robustness, and response speed, overcomes the limitations of traditional technologies in complex environments, and provides stronger guarantees for the application of automated and intelligent systems under various harsh conditions. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 This is an exemplary network architecture used in the autonomous vehicle path planning system based on a large language model in the embodiments of this application;

[0043] Figure 2 This is a flowchart illustrating the autonomous vehicle path planning method based on a large language model in the embodiments of this application;

[0044] Figure 3 This is an exemplary network architecture for the multi-sensor information fusion module in the embodiments of this application;

[0045] Figure 4 This is a schematic diagram of the autonomous vehicle path planning device based on a large language model in the embodiments of this application;

[0046] Figure 5 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0048] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0050] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0051] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0052] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0053] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0054] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0055] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0056] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0057] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0058] Please see Figure 1 The autonomous vehicle path planning method based on a large language model proposed in this application can be implemented based on an autonomous vehicle path planning system based on a large language model. It mainly consists of a data acquisition module, an information fusion module, a path planning module, and a decision execution module.

[0059] In some embodiments, the data acquisition module integrates multiple sensors, including a vision sensor, a LiDAR (Light Detection and Ranging) system, a GPS module, and an inertial measurement unit (IMU). The vision sensor uses a Realsense D455 depth camera to capture image information of the surrounding environment; the LiDAR uses a Mid-360 LiDAR to measure distances to surrounding objects and generate high-precision environmental point clouds; the IMU, built into the LiDAR, integrates a three-axis accelerometer and gyroscope to provide acceleration and angular velocity motion status information of the moving platform, helping to calculate attitude changes; the GPS module can acquire global real-time position information to ensure global positioning accuracy of the path. Millimeter-wave radar is used to detect dynamic obstacles in the environment, providing additional environmental awareness, especially in adverse weather conditions. All sensor data is transmitted via a high-bandwidth interface (CAN bus) to the central processing unit in real time to ensure timeliness and accuracy.

[0060] In some embodiments, the information fusion module utilizes multi-sensor data fusion technology to comprehensively process information from different sensors. This module employs fusion and filtering algorithms to ensure the accuracy and completeness of the information, thereby compensating for potential data gaps and noise interference that may occur with a single sensor in complex environments. The fusion algorithm analyzes visual data, extracts features, and fuses them with LiDAR data to ensure information consistency. The filtering algorithm introduces Kalman filters and particle filters to process noise in sensor data and improve the accuracy of position and attitude estimation. The path planning module uses graph-based path planning algorithms, such as the A* algorithm, combined with the reasoning capabilities of a multimodal large language model to achieve optimal path search. The algorithm can analyze historical data and adjust the plan based on real-time data, handling sudden obstacles or environmental changes to ensure path safety and dynamic replanning capabilities. The multimodal large language model, through training, understands the semantics of different sensor data and can intelligently analyze, process, and filter the collected data, including processing environmental information described in text (such as map information and obstacle descriptions), incorporating this information into path planning considerations. The decision execution module transforms the path planning results into specific execution instructions, controlling the mobile platform to travel along the planned path. The system will monitor environmental changes in real time and dynamically adjust the driving route based on the new data.

[0061] Based on the above exemplary scenario, please refer to Figure 2 In one embodiment of the autonomous vehicle path planning method based on a large language model of this application, the method includes:

[0062] Step S10: Respond to the instruction to perform path planning for the autonomous vehicle, and acquire the traffic scene RGB image, traffic scene depth image, LiDAR point cloud data and inertial measurement unit data of the autonomous vehicle.

[0063] The autonomous vehicle path planning system based on a large language model in autonomous vehicles can respond to instructions for path planning of autonomous vehicles and acquire RGB images of traffic scenes, depth images of traffic scenes, LiDAR point cloud data and inertial measurement unit data of autonomous vehicles.

[0064] Specifically, please refer to Figure 3 An exemplary structure of a multi-sensor information fusion module is as follows: Figure 2 As shown, it is necessary to integrate information from different data sources such as visual sensors, LiDAR, and inertial measurement units (IMU). The core task of this module is to ensure the accuracy and timeliness of data fusion. The visual sensors include RGB-D cameras, and the LiDAR includes Mid-360LiDAR.

[0065] The RGB-D camera acquires RGB images and depth images of different traffic scenarios. Preprocessing includes image calibration and filtering, and camera calibration is performed using a checkerboard method to reduce errors caused by lens distortion. The calibrated camera data can be used to extract depth information from the depth images and perform image blur analysis to ensure that the acquired data is clear and reliable.

[0066] The LiDAR point cloud data can be captured using a Mid-360LiDAR to capture surrounding 3D point cloud information. A voxel grid method is used to downsample the original LiDAR point cloud data, reducing data size and computational complexity. Furthermore, unnecessary point cloud data is filtered out using ROI boxes to reduce storage burden and processing time.

[0067] Inertial measurement unit (IMU) data includes high-frequency linear acceleration and angular velocity, etc., which are used to extract pose information of key frames and perform data calibration. IMU data is reduced for noise and improved for accuracy through Kalman filtering and complementary filtering algorithms. The preprocessed IMU data can be combined with feature data from LiDAR and vision sensors to improve the overall perception accuracy.

[0068] like Figure 2 As shown, the information fusion module proposed in this embodiment includes a modal encoder, a visual encoder that extracts visual features, and generates a feature map. Where N is the number of spatial locations and d is the feature dimension; the point cloud encoder architecture uses the PointNet++ network to process the LiDAR point cloud data in the LiDAR, generating feature vectors of the captured spatial structure.

[0069] Furthermore, the RGB image of the traffic scene is resized to 224×224 pixels, and normalized based on the mean and standard deviation of the ImageNet dataset. Techniques such as random cropping, horizontal flipping, and rotation are used to enhance the data, increase robustness, and prevent model overfitting. The LiDAR point cloud data is voxelized to reduce data redundancy, and k-nearest neighbor statistical filtering is used to remove outliers and noise. The RGB image, depth image, LiDAR point cloud data, and IMU data of the traffic scene are aligned based on timestamps to ensure temporal consistency among the sensor information. Synchronization technology uses a global clock or software clock to ensure that the information fusion module receives data from the same point in time.

[0070] More specifically, a multi-sensor information fusion module is constructed. First, it receives information from different data sources, such as inertial measurement units (IMUs). It then preprocesses and initially filters the data from depth cameras and LiDAR, extracting real-time LiDAR (Light Detection and Ranging) and visual features, and storing the extracted feature information in local mapping. Next, it preprocesses the raw IMU data, extracting pose information from keyframes and high-frequency motion data. Finally, by scanning the context and based on the motion trajectory of the fused radar and camera information, the IMU performs matching and correction to avoid pose drift caused by feature matching errors, thereby maximizing robustness.

[0071] The process involves using an RGB-D camera mounted on a robot to capture videos of different scenes, acquiring RGB and depth images from the raw video frames. Before use, the camera sensor needs to be calibrated because the camera lens distorts the images to some extent. Here, a checkerboard-patterned black-and-white image is used, and the calibration error is calculated by comparing the camera output image with the original image. Blurry images and depth images from different environments are assessed, and clear images are selected. These images are then preprocessed and saved to obtain initial image data.

[0072] The system employs a Mid-360 LiDAR to capture the field of view (FoV) information around the robot. Built-in timers calculate the speed of light and time of flight (ToF) between emitted and received laser beams to obtain the position and state information of surrounding traffic targets. Because the Mid-360 LiDAR's raw point cloud has high resolution and wide coverage, its computational cost is high. Therefore, a voxel grid filter is used for downsampling, and a region of interest (ROI) is used to filter irrelevant point cloud information, reducing computational burden and storage requirements. The remaining position and state information is stored in the processor in the same point cloud format. An inertial measurement unit (IMU) is used to acquire dynamic information about traffic targets or equipment for further filtering and calibration. This reduces attitude errors in keyframes and improves the accuracy of visual loop closure detection. Furthermore, IMU data can be used to address point cloud distortion and rotational drift caused by LiDAR or visual movement; and, when feature matching results from cameras or LiDAR exhibit minor errors, such as feature matching failures due to dynamic environments or local occlusion, IMU data can provide continuous dynamic constraints to prevent error accumulation.

[0073] The data acquisition process needs to consider changes in ambient lighting, irregular terrain, dynamic obstacles, and weather changes to ensure the diversity and richness of the dataset. Visual images are resized and normalized to 224*224 pixels and normalized using the mean and standard deviation of the ImageNet dataset. Data augmentation techniques such as random cropping, flipping, and rotation are used, and color dithering is employed to increase data diversity and prevent overfitting. The LiDAR point cloud data is voxelized to reduce data size and computational complexity, and statistical outlier removal is performed to eliminate stray points. Sensor data is synchronized based on timestamps to ensure temporal consistency between modalities, and calibration parameters provided in the dataset are used to align spatial data from different sensors. Text data is tokenized using a word tagger, and then pre-trained embeddings from models such as Transformer and GPT (including GPT-4) are used to represent the text data.

[0074] Designing a path planning algorithm for an unknown environment requires constructing a semantic 3D traffic scene representation of the environment using an autonomous vehicle, and generating a plan based on this scene using a Large Language Model (LLM). The autonomous vehicle's goal is to find a target location in the environment and create a rich scene representation around the traffic target for downstream planning. The reasoning ability of the LLM is utilized to understand the semantic priorities of the environment, thereby guiding the robot to the target area. At each time stamp, the autonomous vehicle takes RGB images, depth information, and pose as input, and uses a segmentation module to identify objects in the RGB images and generate masks for them. At this point, the LLM acts as a pruning tool, identifying the most important objects in the scene based on the understanding of the environment's semantics, detecting fragments, and pruning unnecessary fragments. Based on a similarity criterion, these detected fragments are converted into a 3D traffic scene representation, providing semantic information for the LLM-based path planning algorithm, where each target is represented by a node. The target node stores information about the object's location, point cloud, the frame in which the target was detected, the target's mask and label in that frame, and semantic description information about the target in the scene. Then, the autonomous vehicle selects a node based on the generated modal weights for the nearest exploration to find the target region. A large language model is combined with path planning, using a modular representation of the 3D traffic scene to determine whether to continue exploring the environment or approach the already detected target. During this process, the autonomous vehicle stores relevant information for each frame in a memory module. If the autonomous vehicle determines that there is no good opportunity to approach the target region in the 3D traffic scene, it will continue exploring the scene and building a 3D traffic scene representation until it reaches the target.

[0075] The autonomous vehicle constructs a modular 3D traffic scene. At the current time step t, it segments and annotates the RGB image frames of the traffic scene, adds target labels, records and stores them, and then converts the target label set into text prompts and corresponding masks. At the same time, it generates 3D object nodes by combining the point cloud information and pose information at the current time step t.

[0076] To enable the Large Language Model (LLM) to effectively process and infer multimodal data, multi-sensor information is converted into text descriptions suitable for the LLM input. Before conversion, the multimodal information requires preprocessing, including data source cleaning and enhancement methods. Based on performance, compatibility, and computational requirements, the GPT-4 model is chosen as the LLM architecture to ensure compatibility between the embeddings generated by the LLM and those generated by the sensor encoders. An API is created to facilitate communication between the LLM and the fusion module, including data exchange protocols and handling data serialization / deserialization to ensure the LLM output is correctly interpreted by the fusion module. Finally, the feedback signals from the LLM are integrated into the parameter adjustment mechanism of the fusion module. This allows for refining the weight module and updating the fusion strategy to improve system performance.

[0077] The modality weighting mechanism in the decision-making model based on the Large Language Model (LLM) is based on the attention weights of the target node generated by the environment, including attention fusion and cross-modal conversion. Based on the LLM's reasoning process, different cue structures, information input, and contextual cues, the LLM's response to modality weighting and policy selection is optimized. Simultaneously, ablation experiments are conducted by disabling or modifying information from some sensors in the LLM-based decision-making module to evaluate the impact of partial modal information loss on overall performance. Sensitivity analysis is performed by changing the LLM's learning rate and attention mechanism to determine the LLM's sensitivity to parameters, ensuring stability and efficiency.

[0078] The path planning module utilizes goal orientation and environment mapping to construct obstacle maps, boundary maps, and target maps for the current scene. In the initial scene state, the target map and boundary map are identical. After processing multimodal information to construct the current scene, the target map is constructed based on the robot's proximity to the target area selected by the Large Language Model (LLM model). The target map and obstacle map are then used as the basis for the path planning algorithm's selection.

[0079] In some embodiments, after acquiring the RGB image of the traffic scene, the depth image of the traffic scene, the LiDAR point cloud data, and the inertial measurement unit data of the autonomous vehicle, the following steps are included:

[0080] Step S101: Using a Bayesian filter, the pose estimation is updated through probabilistic reasoning based on the RGB image of the traffic scene, the depth image of the traffic scene, the lidar point cloud data, and the inertial measurement unit data.

[0081] Step S102: Continuously update the local map based on the updated pose estimation. When sufficient pose information is obtained, expand the local map into a global map to construct a complete 3D traffic scene representation.

[0082] Step S20: Extract features from the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the lidar, and the data of the inertial measurement unit to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories.

[0083] After acquiring RGB images, depth images, LiDAR point cloud data, and inertial measurement unit data of the traffic scene from the autonomous vehicle, feature extraction is performed on the RGB images, depth images, LiDAR point cloud data, and inertial measurement unit data to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories.

[0084] In some embodiments, the step of extracting features from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information includes:

[0085] Step S201: Use a preset convolutional neural network to extract low-level feature information corresponding to each traffic target in the traffic scene RGB image, wherein the low-level feature information includes the edge, texture and contour of the traffic target;

[0086] Step S202: Call a preset deep convolutional neural network to extract features from the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative position between each traffic target. Use a preset feature extraction algorithm to extract the local features of each traffic target in the traffic scene depth image.

[0087] Step S203: Use a pre-trained semantic segmentation network to perform semantic segmentation on the lidar point cloud data, and fuse the low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target, and the local features of each traffic target to determine the different traffic targets in the traffic scene and their corresponding target categories.

[0088] Specifically, after the preprocessing stage, features are extracted from the RGB images, depth images, and LiDAR point cloud data of the traffic scene and input into the feature extraction module to construct a 3D traffic scene representation. Features in the RGB images can be extracted using Convolutional Neural Networks (CNNs) and deep learning techniques. Features in the depth images can be supplemented by deep learning networks, such as using a hybrid feature extraction method combining Deep Convolutional Networks (DCNNs) and feature extraction algorithms. LiDAR provides point cloud data for the observation model. Point cloud features are distinguished by ground segmentation and semantic segmentation networks to differentiate different traffic targets and their corresponding target categories. High-frequency angular velocity and acceleration information can be extracted from IMU data for attitude estimation and motion prediction. The predicted pose of the Bayesian filter is corrected using a system fusion model. The mapping algorithm uses the corrected predicted pose to create a local map. When the optimal pose is obtained, the local map is updated to a global map. The global map is then Gaussian blurred to construct a new observation model, thus forming a complete closed loop for the 3D traffic scene.

[0089] In some embodiments, the basic network architecture of the semantic segmentation network includes PointNet++ network or RangeNet++ network, etc.; the feature extraction algorithm includes SIFT feature extraction algorithm or ORB feature extraction algorithm, etc.

[0090] Step S30: Call the preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information, wherein the natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it.

[0091] Feature extraction is performed on the RGB image of the traffic scene, the depth image of the traffic scene, the LiDAR point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information. Then, a preset large language model is invoked to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it. The traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles, or buildings, etc. The motion state includes the current position, yaw angle, speed, and acceleration, etc. The large language model includes the Transformer model or the GPT model, etc.

[0092] In some embodiments, the step of invoking a preset large language model to determine the motion state of the autonomous vehicle and a natural language description of the surrounding traffic targets based on the multimodal traffic scene information includes:

[0093] Step S301: Call the preset large language model, perform time alignment and spatial alignment on the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data and the inertial measurement unit data and fuse them to generate a context embedding that includes the motion state of the autonomous vehicle and the traffic targets around it.

[0094] Step S302: Generate a natural language description of the motion state of the autonomous driving system and the surrounding traffic targets based on the fused context embedding.

[0095] Specifically, transmitting the multimodal traffic scene information to the large language model requires converting it into a description suitable for text input. This involves using natural language to describe the motion state of the autonomous vehicle and the surrounding traffic targets. For example, the natural language description of the traffic targets around the autonomous vehicle could be expressed as "There is a car 1 meter ahead, and an obstacle 2 meters to the left," and so on. Similarly, the natural language description of the motion state of the autonomous vehicle could be expressed as "The robot is moving forward at a speed of 1 meter per second," and so on.

[0096] Data encoding employs a wordpiece to segment the text and generate semantic embeddings. These semantic embeddings are then aligned and fused with other sensor data. The text encoder encodes the text input based on a converter model. In contextual embeddings, the fusion module integrates features from different modalities using a large language model (LLM), connecting the modal features. Where M is the number of modalities; the Transformer layer utilizes attention mechanisms and feedforward networks to simulate complex dependencies and generate unified data. express.

[0097] Step S40: Using the attention mechanism in the large language model, the attention weight of the target node is determined based on the natural language description of the traffic target. A preset path planning algorithm is used to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle.

[0098] After determining the motion state of the autonomous vehicle and the natural language description of the surrounding traffic targets based on the multimodal traffic scene information by calling a preset large language model, the attention mechanism in the large language model is used to determine the attention weight of the target node based on the natural language description of the traffic target. Then, a preset path planning algorithm is used to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle. The path planning algorithm includes A* algorithm, Dijkstra algorithm, or RRT algorithm, etc.

[0099] In some embodiments, the steps of using the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, include:

[0100] Step S401: Generate a natural language description of the motion state of the autonomous vehicle and the surrounding traffic targets based on a preset large language model for path planning.

[0101] Step S402: Construct a 3D traffic scene representation at a certain time step. Based on the attention mechanism in the large language model, assign attention weights to the target nodes according to the target mask generated by segmenting the RGB image of the traffic scene and the natural language description of the traffic target, so as to determine the attention weight of the target node.

[0102] Step S403: Using a preset path planning algorithm, explore the most feasible path planning based on the attention weight of the target node. Upon receiving new sensor data and feedback from the large language model, update the 3D traffic scene representation and recalculate the target position and driving path to determine the optimal driving path for the autonomous vehicle.

[0103] Specifically, the GPT-4 model can be chosen as the architecture for the Large Language Model (LLM model), which does not limit this application. Data communication and interaction can be achieved through an Application Programming Interface (API), and the API is designed accordingly. The API design considers serialization and deserialization to ensure smooth data transfer between modules. The API supports data transmission protocols such as JSON or Protobuf. Model optimization involves experimentally selecting the optimal learning rate and attention mechanism, and tuning parameters to improve responsiveness to multimodal inputs. Ablation experiments are used to evaluate system performance when some data is missing and to analyze the sensitivity of the Large Language Model (LLM model) to different modality weights.

[0104] Path planning is performed based on the multimodal traffic scene semantic representation generated by the Large Language Model (LLM model). A 3D traffic scene representation is constructed at a certain time step. Based on the attention mechanism in the LLM model, attention weights are assigned to target nodes according to the target mask generated by segmenting the RGB image of the traffic scene and the natural language description of the traffic target. A preset path planning algorithm is then used to explore the most feasible path based on the attention weights of the target nodes. Upon receiving new sensor data and feedback from the LLM model, the 3D traffic scene representation is updated, and the target position and driving path are recalculated to determine the optimal driving path for the autonomous vehicle.

[0105] In a further embodiment, the decision results of the Large Language Model (LLM) are integrated into the path planning module. Weight adjustment optimizes the weight parameters of the traffic scene RGB image, traffic scene depth image, LiDAR point cloud data, and the motion state of the autonomous vehicle and its surrounding traffic targets based on multimodal inputs, achieving weight allocation across modal information. Policy selection is based on a pruning algorithm of the LLM model, identifying key objects and ignoring irrelevant parts based on environmental information. Sensitivity scoring evaluates the model's stability and efficiency under different configurations by adjusting the learning rate and attention mechanism.

[0106] The path planning module constructs obstacle maps, boundary maps, and target maps for path calculation. Initially, the boundary map and target map are identical, and the target region is updated as exploration progresses. Path calculation employs heuristic algorithms (such as A algorithm and Dijkstra's algorithm) or optimization algorithms (such as RRT algorithm and D-Lite algorithm) to generate paths, combining semantic understanding and scene analysis from the LLM model to determine the optimal driving path. Furthermore, during real-time path planning execution, changes in the traffic environment are monitored, and the path is recalculated to avoid dynamic obstacles.

[0107] In some embodiments, the pseudocode representation of the autonomous vehicle path planning method based on a large language model of this application is as follows:

[0108]

[0109] As can be seen from the above embodiments, compared with the prior art, the present application addresses the problems of pure vision path planning schemes being affected by light and occlusion, and the perception of the environment relying on lidar being affected by severe weather and noise generated by the propagation of the laser beam. The present application has, but is not limited to, the following beneficial effects:

[0110] Firstly, this application integrates information from multiple sensors, including vision, LiDAR, and inertial measurement unit (IMU), enabling the system to achieve more comprehensive and accurate environmental perception. The complementarity of different sensors allows the system to maintain high perception accuracy even in complex or harsh environments. For example, vision sensors can provide rich scene information, while LiDAR can accurately perceive the distance and shape of objects, and the IMU provides dynamic motion information. Through the fusion of this information, the system's understanding of the environment becomes more precise, thereby significantly reducing decision-making errors caused by the reliance on a single sensor and improving the reliability and security of overall decision-making.

[0111] Secondly, this application employs a large language model, which enables rapid analysis and processing of massive amounts of traffic environment data. Since route planning typically requires real-time responses to complex environmental changes, combining it with a large language model can effectively improve the system's computational efficiency and response speed. The advantages of large language models in language understanding and reasoning allow the system to quickly extract useful information from multiple sensors and rapidly generate reasonable route planning decisions based on the current context. This efficient data processing capability ensures that the system reacts promptly in rapidly changing environments, avoiding safety hazards caused by delays.

[0112] Thirdly, the route planning system of this application possesses a high degree of adaptability, capable of dynamically adjusting the route based on real-time perceived environmental information. Especially in complex outdoor environments, factors such as obstacles, road conditions, and climate change constantly influence route selection. The large language model not only combines multi-sensor information for real-time data analysis but also optimizes route selection through learning and reasoning, ensuring that route planning can be adjusted promptly under different environmental conditions to guarantee driving safety and efficiency. For example, when encountering temporary obstacles or sudden weather changes, the system can recalculate a new route in the shortest possible time, preventing accidents.

[0113] Fourth, this application significantly enhances the robustness of path planning through multimodal information fusion technology. Different types of sensors can complement each other; if one sensor fails or is interfered with, the others can continue to provide valid data, thereby ensuring the overall stability and reliability of the system. This robustness of multi-sensor fusion enables the system to provide efficient and accurate path planning decisions even under complex environmental conditions (such as low light, severe weather, or dense obstacle environments), reducing safety risks caused by single sensor failure or interference.

[0114] Fifth, this application overcomes the limitations of traditional path planning algorithms. Traditional path planning methods often rely on a single sensor or rule engine, which is easily limited by factors such as environmental complexity and sensor accuracy. This application, however, overcomes these limitations by combining data from multiple sensors and the processing power of a large language model. The system can not only handle dynamically changing and complex environments but also possesses strong generalization capabilities, adapting to different scenarios and tasks and providing more intelligent and flexible path planning.

[0115] Furthermore, by combining multimodal information fusion with a large language model, this application provides an innovative path planning method that significantly improves the perception, decision-making, and adaptive capabilities of path planning systems in complex and dynamic environments, enhances the system's reliability, robustness, and response speed, overcomes the limitations of traditional technologies in complex environments, and provides stronger guarantees for the application of automated and intelligent systems under various harsh conditions.

[0116] Please see Figure 4 This application provides a path planning device for autonomous vehicles based on a large language model, comprising a data acquisition module 1100, a feature extraction module 1200, a natural language description module 1300, and a path planning module 1400, all for the purposes of this application. The data acquisition module 1100 is configured to acquire RGB images of the traffic scene, depth images of the traffic scene, LiDAR point cloud data, and inertial measurement unit data of the autonomous vehicle in response to a path planning instruction for the autonomous vehicle. The feature extraction module 1200 is configured to extract features from the RGB images of the traffic scene, the depth images of the traffic scene, the LiDAR point cloud data, and the inertial measurement unit data, respectively, to construct multimodal traffic scene information, wherein the multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories. The natural language description module 1300 is configured to... The system is configured to invoke a preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it. The path planning module 1400 is configured to use the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target location, so as to complete the path planning of the autonomous vehicle.

[0117] Based on any embodiment of this application, please refer to Figure 5Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 5 The diagram shows the internal structure of a computer device. This computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, they enable the processor to implement an autonomous vehicle path planning method based on a large language model. The processor of this computer device provides computational and control capabilities, supporting the operation of the entire computer device. The memory of this computer device may store computer-readable instructions. When these computer-readable instructions are executed by the processor, they enable the processor to execute the autonomous vehicle path planning method based on a large language model of this application. The network interface of this computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In this embodiment, the processor is used to execute... Figure 4 The memory stores the specific functions of each module, and stores the program code and various data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules in the autonomous vehicle path planning device based on a large language model of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0119] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the autonomous vehicle path planning method based on a large language model as described in any embodiment of this application.

[0120] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the autonomous vehicle path planning method based on a large language model as described in any embodiment of this application.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0122] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0123] In summary, by combining multimodal information fusion with a large language model, this application provides an innovative path planning method that significantly improves the perception, decision-making, and adaptive capabilities of path planning systems in complex and dynamic environments. It enhances the reliability, robustness, and response speed of the system, overcomes the limitations of traditional technologies in complex environments, and provides stronger protection for the application of automated and intelligent systems under various harsh conditions.

Claims

1. A path planning method for autonomous vehicles based on a large language model, characterized in that, include: In response to instructions for path planning of autonomous vehicles, the system acquires RGB images of traffic scenes, depth images of traffic scenes, LiDAR point cloud data, and inertial measurement unit data of autonomous vehicles. Feature extraction is performed on the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the lidar, and the data of the inertial measurement unit to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories. A preset large language model is invoked to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it. The attention mechanism in the large language model is used to determine the attention weight of the target node based on the natural language description of the traffic target. A preset path planning algorithm is then used to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle.

2. The autonomous vehicle path planning method based on a large language model according to claim 1, characterized in that, The steps of extracting features from the RGB image of the traffic scene, the depth image of the traffic scene, the lidar point cloud data, and the inertial measurement unit data to construct multimodal traffic scene information include: A pre-defined convolutional neural network is used to extract low-level feature information corresponding to each traffic target in the RGB image of the traffic scene. The low-level feature information includes the edges, textures, and contours of the traffic targets. A preset deep convolutional neural network is invoked to extract features from the traffic scene depth image to capture the spatial layout of each traffic target in the traffic scene depth image and the relative positions between each traffic target. A preset feature extraction algorithm is used to extract the local features of each traffic target in the traffic scene depth image. A pre-trained semantic segmentation network is used to perform semantic segmentation on the lidar point cloud data. The low-level feature information corresponding to each traffic target, the spatial layout of each traffic target, the relative position between each traffic target, and the local features of each traffic target are fused to determine the different traffic targets in the traffic scene and their corresponding target categories.

3. The autonomous vehicle path planning method based on a large language model according to claim 1, characterized in that, After acquiring RGB images of the traffic scene, depth images of the traffic scene, LiDAR point cloud data, and inertial measurement unit data from the autonomous vehicle, the process includes: A Bayesian filter is used to update the pose estimation based on the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the LiDAR, and the data of the inertial measurement unit through probabilistic reasoning. The local map is continuously updated based on the updated pose estimation. When sufficient pose information is obtained, the local map is expanded into a global map to construct a complete 3D traffic scene representation.

4. The autonomous vehicle path planning method based on a large language model according to claim 1, characterized in that, The steps of invoking a preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the surrounding traffic targets based on the multimodal traffic scene information include: The preset large language model is invoked to perform time and space alignment and fusion of the features extracted from the traffic scene RGB image, the traffic scene depth image, the lidar point cloud data and the inertial measurement unit data, so as to generate a context embedding that includes the motion state of the autonomous vehicle and the traffic targets around it. A natural language description of the motion state of the autonomous vehicle and its surrounding traffic targets is generated based on the fused context embedding.

5. The autonomous vehicle path planning method based on a large language model according to claim 3, characterized in that, The steps of using the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and using a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, include: Path planning is performed by generating natural language descriptions of the motion state of the autonomous vehicle and the surrounding traffic targets based on a preset large language model. A 3D traffic scene representation is constructed at a certain time step. Based on the attention mechanism in the large language model, attention weights are assigned to the target nodes according to the target mask generated by segmenting the RGB image of the traffic scene and the natural language description of the traffic target, so as to determine the attention weight of the target node. A preset path planning algorithm is used to explore the most feasible path planning based on the attention weight of the target node. Whenever new sensor data and large language model feedback are received, the 3D traffic scene representation is updated and the target position and driving path are recalculated to determine the optimal driving path of the autonomous vehicle.

6. The autonomous vehicle path planning method based on a large language model according to any one of claims 1 to 5, characterized in that, The traffic targets include vehicles, pedestrians, lane lines, passable areas, obstacles, or buildings; the motion states include current position, yaw angle, speed, and acceleration.

7. The autonomous vehicle path planning method based on a large language model according to claim 2, characterized in that, The large language model includes the Transformer model or the GPT model; the basic network architecture of the semantic segmentation network includes the PointNet++ network or the RangeNet++ network; the feature extraction algorithm includes the SIFT feature extraction algorithm or the ORB feature extraction algorithm; and the path planning algorithm includes the A* algorithm, the Dijkstra algorithm, or the RRT algorithm.

8. A path planning device for autonomous vehicles based on a large language model, characterized in that, include: The data acquisition module is configured to respond to instructions for path planning of autonomous vehicles by acquiring RGB images of traffic scenes, depth images of traffic scenes, LiDAR point cloud data, and inertial measurement unit data of autonomous vehicles. The feature extraction module is configured to extract features from the RGB image of the traffic scene, the depth image of the traffic scene, the point cloud data of the lidar, and the data of the inertial measurement unit, respectively, to construct multimodal traffic scene information. The multimodal traffic scene information is constructed from the motion state of the autonomous vehicle, multiple different traffic targets, and their corresponding target categories. The natural language description module is configured to call a preset large language model to determine the motion state of the autonomous vehicle and the natural language description of the traffic targets around it based on the multimodal traffic scene information. The natural language description is used to describe the motion state of the autonomous vehicle and the location information of the traffic targets around it. The path planning module is configured to use the attention mechanism in the large language model to determine the attention weight of the target node based on the natural language description of the traffic target, and use a preset path planning algorithm to determine the optimal driving path based on the attention weight of the target node, the motion state of the autonomous vehicle, and the target position, so as to complete the path planning of the autonomous vehicle.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images