Path planning method, model training method, device, equipment and medium

Abstract

The invention discloses a path planning method, a model training method and device, equipment and a medium, and relates to the technical field of computers. The method comprises the following steps: acquiring to-be-processed own vehicle driving information, traffic participant driving information and a bird's-eye view feature map corresponding to own vehicle sensing information; inputting the own vehicle driving information, the traffic participant driving information and the aerial view feature map into a pre-trained path planning model so as to obtain fused feature coding information based on a feature coding network of the path planning model, the own vehicle driving information, the traffic participant driving information and the aerial view feature map; performing conversion processing on the fusion feature coding information to obtain the fusion feature coding information after conversion processing; obtaining an output result of the path planning model based on a large language model network of the path planning model and the converted fusion feature coding information; and obtaining path planning information of the vehicle based on an output result of the path planning model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as intelligent transportation and autonomous driving, and in particular to a path planning method, model training method, device, equipment and medium. Background Art

[0002] Generally, the path planning system is one of the core systems of autonomous driving technology. Its main task is to generate a better driving route for the autonomous driving vehicle based on perception information, prediction information, and high-precision map information.

[0003] At present, the path planning system in the related art is usually divided into three sub-modules: behavior decision, path planning and motion planning. Among them, the path planning module usually needs to take into account the high-precision map information and the output of the behavior decision module to generate a global path from the current position of the vehicle to the target position. However, the path planning system in the related art is overly dependent on high-precision maps. Due to the difficulty and high cost of creating, updating and maintaining high-precision maps, the pressure of data calculation and processing in the path planning process is relatively high. Summary of the invention

[0004] The present application provides a path planning method, a model training method, an apparatus, a device and a medium, which solve the problems such as poor performance of the path planning system. The technical solution is as follows:

[0005] In a first aspect, a path planning method is provided, the method comprising:

[0006] Acquire the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle;

[0007] Inputting the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map into a pre-trained path planning model to obtain fused feature coding information based on a feature coding network of the path planning model and the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map;

[0008] Performing conversion processing on the fused feature coding information to obtain converted fused feature coding information;

[0009] Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained;

[0010] Based on the output result of the path planning model, the path planning information of the vehicle is obtained.

[0011] In a possible implementation, the feature coding network includes a vehicle feature coding network, a bird's-eye view feature coding network, and a traffic participant feature coding network. The feature coding network based on the path planning model and the vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map obtains fused feature coding information, including:

[0012] Based on the vehicle feature coding network and the vehicle driving information, obtaining vehicle driving feature coding information;

[0013] Based on the bird's-eye view feature coding network and the bird's-eye view feature map, obtaining bird's-eye view feature map coding information;

[0014] Based on the traffic participant feature coding network and the traffic participant driving information, obtaining traffic participant driving feature coding information;

[0015] The vehicle driving characteristic coding information, the bird's-eye view characteristic coding information, and the traffic participant driving characteristic coding information are fused to obtain fused characteristic coding information.

[0016] In a possible implementation, the bird's-eye view feature encoding network includes a spatial feature encoder and a position feature encoder, and obtaining the bird's-eye view feature map encoding information based on the bird's-eye view feature encoding network and the bird's-eye view feature map includes:

[0017] Using the spatial feature encoder, performing spatial feature encoding processing on the bird's-eye view feature image to obtain a result of the spatial feature encoding processing;

[0018] Using the position feature encoder, performing position feature encoding processing on the bird's-eye view feature image to obtain a result of the position feature encoding processing;

[0019] The result of the spatial feature encoding process and the result of the position feature encoding process are added together to obtain the bird's-eye view feature map encoding information.

[0020] In a possible implementation, the path planning model includes a connector, and the converting process of the fused feature encoding information to obtain the converted fused feature encoding information includes:

[0021] The connector is used to convert the fused feature coding information to obtain the converted fused feature coding information.

[0022] In a possible implementation, the path planning model includes a decoding network, the large language model network based on the path planning model and the fused feature encoding information after the conversion process, and obtaining the output result of the path planning model includes:

[0023] Inputting the converted fusion feature encoding information into the large language model network to obtain an output result of the large language model network;

[0024] Using the decoding network, decoding the output result of the large language model network;

[0025] Based on the result of the decoding process, an output result of the path planning model is obtained.

[0026] In a second aspect, a method for training a path planning model is provided, wherein the path planning model to be trained includes a feature encoding network and a large language model network, and the method includes:

[0027] Acquire training sample data, the training sample data including sample ego-vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego-vehicle perception information;

[0028] Inputting the training sample data into a path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data;

[0029] Performing conversion processing on the fused feature coding information to obtain converted fused feature coding information;

[0030] Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained;

[0031] Based on the training sample data and the output result of the path planning model, the path planning model to be trained is updated and trained to obtain a trained path planning model.

[0032] In a third aspect, a path planning device is provided, the device comprising:

[0033] An acquisition unit, used to acquire the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle;

[0034] an encoding unit, configured to input the self-vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map into a pre-trained path planning model, so as to obtain fused feature encoding information based on a feature encoding network of the path planning model and the self-vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map;

[0035] A conversion unit, used for performing conversion processing on the fused feature coding information to obtain the fused feature coding information after conversion processing;

[0036] An output unit, used for obtaining an output result of the path planning model based on the large language model network of the path planning model and the fused feature encoding information after the conversion process;

[0037] The obtaining unit is used to obtain the path planning information of the vehicle based on the output result of the path planning model.

[0038] In a fourth aspect, a training device for a path planning model is provided, wherein the path planning model to be trained includes a feature encoding network and a large language model network, and the device includes:

[0039] A first acquisition unit is used to acquire training sample data, wherein the training sample data includes sample ego vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego vehicle's perception information;

[0040] A first obtaining unit, configured to input the training sample data into a path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data;

[0041] A first conversion unit is used to convert the fused feature coding information to obtain the converted fused feature coding information;

[0042] A second obtaining unit is used to obtain an output result of the path planning model based on the large language model network of the path planning model and the fused feature encoding information after the conversion process;

[0043] The first training unit is used to update the path planning model to be trained based on the training sample data and the output result of the path planning model to obtain a trained path planning model.

[0044] In a fifth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspects and any possible implementation manner.

[0045] In a sixth aspect, an electronic device is provided, including:

[0046] at least one processor; and

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0049] In a seventh aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0050] The beneficial effects of the technical solution provided by this application include at least:

[0051] It can be seen from the above technical scheme that, on the one hand, the embodiment of the present application can obtain the self-vehicle driving information to be processed, the traffic participant driving information and the bird's-eye view feature map corresponding to the perception information of the self-vehicle, and then the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map can be input into a pre-trained path planning model to obtain fused feature coding information based on the feature coding network of the path planning model and the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map, convert the fused feature coding information to obtain the converted fused feature coding information, and the large language model network of the path planning model and the converted fused feature coding information. The feature coding information is combined to obtain the output result of the path planning model, so that the path planning information of the vehicle can be obtained based on the output result of the path planning model. Since the pre-trained path planning model can be directly used, the path planning information of the vehicle can be obtained only based on the bird's-eye view feature map corresponding to the driving information of the vehicle, the driving information of the traffic participants and the perception information of the vehicle. While ensuring the accuracy of the path planning result, the corresponding path planning result can be directly output based on the end-to-end planning model, avoiding the use of high-precision map related information, reducing computing power overhead, optimizing planning performance and efficiency, and thus improving the reliability of path planning of autonomous driving vehicles.

[0052] It can be seen from the above technical solution that, on the other hand, the embodiment of the present application can obtain training sample data, wherein the training sample data includes sample vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the perception information of the vehicle, and then the training sample data can be input into the path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data, convert the fused feature coding information to obtain the converted fused feature coding information, and obtain the output result of the path planning model based on the large language model network of the path planning model and the converted fused feature coding information, so as to be able to The path planning model to be trained can be updated and trained based on the training sample data and the output result of the path planning model to obtain a trained path planning model. Since an end-to-end path planning model can be trained based on a sample bird's-eye view feature map corresponding to sample ego-vehicle driving information, sample traffic participant driving information and ego-vehicle perception information, a more accurate and effective path planning model can be obtained without using high-precision map data. While ensuring the performance and effectiveness of the path planning model, the amount of data to be processed by the model is reduced, and the processing computing power overhead of the overall path planning is reduced, thereby ensuring the reliability and performance of the path planning of the autonomous driving vehicle.

[0053] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 It is a flowchart of a path planning method provided by an embodiment of the present application;

[0056] Figure 2 is a flowchart of a training method for a path planning model provided by another embodiment of the present application;

[0057] Figure 3 is a flowchart of a path planning method provided by another embodiment of the present application;

[0058] Figure 4 It is a schematic diagram of the architecture of an application scenario of a path planning method provided by another embodiment of the present application;

[0059] Figure 5A is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a scenario where a vehicle starts and bypasses a parked vehicle in front;

[0060] Figure 5B is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a vehicle turning scenario;

[0061] Figure 5C is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a scenario where a vehicle passes through a construction area;

[0062] Figure 6 is a structural block diagram of a path planning device provided by another embodiment of the present application;

[0063] Figure 7 is a structural block diagram of a training device for a path planning model provided by another embodiment of the present application;

[0064] Figure 8 It is a block diagram of an electronic device used to implement the path planning method and model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0067] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0068] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0069] Please refer to Figure 1 , which shows a flow chart of a path planning method provided by an embodiment of the present application. The path planning method may specifically include:

[0070] Step 101: Obtain the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle.

[0071] Step 102: input the vehicle driving information, the traffic participant driving information and the bird's-eye view feature map into a pre-trained path planning model to obtain fused feature coding information based on the feature coding network of the path planning model and the vehicle driving information, the traffic participant driving information and the bird's-eye view feature map.

[0072] Step 103: convert the fused feature coding information to obtain the converted fused feature coding information.

[0073] Step 104: Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained.

[0074] Step 105: Obtain the path planning information of the vehicle based on the output result of the path planning model.

[0075] It should be noted that the vehicle driving data may include but is not limited to the position, speed, acceleration, direction, etc. of the autonomous driving vehicle.

[0076] It should be noted that the bird's-eye view feature map can be a bird's-eye view feature map determined based on perception information. The bird's-eye view feature map can be rasterized perception information generated based on a bird's-eye view (BEV) spatial representation. The bird's-eye view feature map can be directly obtained from an upstream perception module. The bird's-eye view feature map can include but is not limited to drivable area information, route information, and occupancy information.

[0077] It should be noted that the driving information of traffic participants may include but is not limited to the location, speed, acceleration, direction, etc. of traffic participants. Traffic participants may be other traffic participants in the driving environment of the vehicle. Traffic participants may include motor vehicles, non-motor vehicles, pedestrians, traffic cones, etc.

[0078] It should be noted that the output result of the path planning model may include the planned position information, speed information, and orientation information of the ego vehicle. Here, a planned trajectory may be formed based on the planned position information, speed information, and orientation information of the ego vehicle, and the planned trajectory may be used as the path planning information of the ego vehicle.

[0079] It should be noted that part or all of the execution subject of steps 101 to 105 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a path planning platform on the network side, etc. This embodiment does not specifically limit this.

[0080] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0081] In this way, the input driving information of the own vehicle, the driving information of the traffic participants and the bird's-eye view feature map corresponding to the perception information of the own vehicle can be planned and processed directly based on the pre-trained path planning model to obtain the path planning information of the own vehicle. While ensuring the accuracy of the path planning results, the corresponding path planning results can be directly output based on the end-to-end planning model, which reduces the computing power overhead, optimizes the planning performance and efficiency, and thus improves the reliability of the path planning of the autonomous driving vehicle.

[0082] Moreover, here, the path planning process does not involve high-precision map data, which can further reduce the amount of data to be processed by the model. While ensuring the performance and effectiveness of the path planning model, it can reduce the processing computing overhead of the overall path planning, thereby improving the efficiency of path planning.

[0083] Optionally, in a possible implementation of this embodiment, the feature coding network may include a self-vehicle feature coding network, a bird's-eye view feature coding network, and a traffic participant feature coding network. In step 102, first, based on the self-vehicle feature coding network and the self-vehicle driving information, the self-vehicle driving feature coding information can be obtained. Secondly, based on the bird's-eye view feature coding network and the bird's-eye view feature map, the bird's-eye view feature map coding information can be obtained. Thirdly, based on the traffic participant feature coding network and the traffic participant driving information, the traffic participant driving feature coding information can be obtained. Thirdly, the self-vehicle driving feature coding information, the bird's-eye view feature map coding information, and the traffic participant driving feature coding information are fused to obtain fused feature coding information.

[0084] In this implementation, the bird's-eye view feature encoding network may include a spatial feature encoder and a position feature encoder.

[0085] In a specific implementation process of this implementation mode, first, the spatial feature encoder can be used to perform spatial feature encoding processing on the bird's-eye view feature map to obtain the result of the spatial feature encoding processing. Secondly, the position feature encoder can be used to perform position feature encoding processing on the bird's-eye view feature map to obtain the result of the position feature encoding processing. Thirdly, the result of the spatial feature encoding processing and the result of the position feature encoding processing can be added to obtain the bird's-eye view feature map encoding information.

[0086] In this implementation, the spatial feature encoder may be an encoder based on a convolutional neural network (CNN). The spatial feature encoder may be used to process the spatial feature information in the bird's-eye view feature map.

[0087] In a specific implementation process of this implementation method, the bird's-eye view feature map can be input into a CNN-based encoder, and the result of the spatial feature encoding processing of the bird's-eye view feature map can be output.

[0088] Here, the spatial feature information in the bird's-eye view feature map may include edge information, shape information, and the like.

[0089] In this implementation, the position feature encoding may be an encoder based on a multi-layer perceptron. The position feature encoding may be used to process the position information in the bird's-eye view feature map.

[0090] In another specific implementation process of this implementation, the bird's-eye view feature map may be input into a position feature encoding based on a multi-layer perceptron, and a result of the position feature encoding processing of the bird's-eye view feature map may be output.

[0091] Here, the location feature information in the bird's-eye view feature map may include coordinate information and the like.

[0092] In this way, the input bird's-eye view feature map can be encoded by using the spatial feature encoder and the position feature encoder, respectively, to obtain the corresponding spatial feature encoding processing results and the position feature encoding processing results, so as to obtain more accurate bird's-eye view feature map encoding information, which can further improve the accuracy of the bird's-eye view feature map encoding information.

[0093] In this implementation, the vehicle feature encoding network may include a vehicle driving feature encoder. The vehicle driving feature encoder may be an encoder based on a multi-layer perceptron. The vehicle driving feature encoder may be used to process the driving information of the vehicle. The driving information of the vehicle, i.e., the vehicle itself, may include the position, speed, acceleration, direction, etc. of the vehicle in the current frame and the historical frame of the path planning.

[0094] In this implementation, the traffic participant feature encoding network may include a traffic participant driving feature encoder. The traffic participant driving feature encoder may be an encoder based on a multi-layer perceptron. The traffic participant driving feature encoder may be used to process the driving information of the traffic participant. The driving information of the traffic participant may include the position, speed, acceleration, direction, etc. of the current frame and the historical frame of the path planning.

[0095] In this implementation, the path planning model may further include a concatenation network, which may be a concatenation layer (Concat) network.

[0096] In another specific implementation process of this implementation method, the splicing network is used to fuse the vehicle driving feature coding information, the bird's-eye view feature coding information, and the traffic participant driving feature coding information to obtain fused feature coding information.

[0097] Optionally, in a possible implementation of this embodiment, the path planning model may include a connector. In step 103, the fused feature coding information may be converted using the connector to obtain the converted fused feature coding information.

[0098] In this implementation, a connector can be used to convert the fused feature encoding information into information in the form of tokens to be input into the large language model network. The connector can be based on a connector of a multi-layer perceptron structure.

[0099] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the path planning method of this embodiment. For detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0100] Optionally, in a possible implementation of this embodiment, the path planning model includes a decoding network. In step 104, first, the fused feature encoding information after the conversion process can be input into the large language model network to obtain the output result of the large language model network. Secondly, the output result of the large language model network is decoded using the decoding network. Thirdly, based on the result of the decoding process, the output result of the path planning model is obtained.

[0101] In this implementation, the decoding network may include a decoder. The decoder may be a decoder based on a transform structure. Preferably, the decoder may be used to decode the path information of the output result of the large language model network.

[0102] In this implementation, the large language model network may be a network based on a large language model (LLM). The large language model network may include any of large language models with different parameter amounts.

[0103] Exemplarily, the large language model network may include any of the models with 7 billion (B), 13B, 34B, and 70B parameters. The large language model network may also include, but is not limited to, a large language model based on internlm, a large language model based on llama, a large language model based on vicuna, and the like.

[0104] In a specific implementation process of this implementation method, the fused feature encoding information after the conversion is input into the large language model network, feature learning and information aggregation are performed through the attention mechanism and fully connected layer of the large language model network, and the results of feature learning and information aggregation are passed through the autoregressive layer to obtain the output result of the large language model network.

[0105] In this way, the reliability and accuracy of the model planning results can be further improved by utilizing the large language model network to plan and identify the fused feature encoding information.

[0106] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the path planning method of this embodiment. For detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0107] Figure 2 is a flow chart of a training method for a path planning model provided by another embodiment of the present application, such as Figure 2 The path planning model to be trained may include a feature encoding network and a large language model network, and the method may specifically include:

[0108] Step 201: Acquire training sample data, where the training sample data includes sample ego-vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego-vehicle's perception information.

[0109] Step 202: input the training sample data into the path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data.

[0110] Step 203: convert the fused feature coding information to obtain converted fused feature coding information.

[0111] Step 204: Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained.

[0112] Step 205: Based on the training sample data and the output result of the path planning model, update the path planning model to be trained to obtain a trained path planning model.

[0113] It should be noted that the trained path planning model can be used in the path planning method in the aforementioned embodiment.

[0114] It should be noted that the sample vehicle driving information may include but is not limited to the position, speed, acceleration, direction, etc. of the autonomous driving vehicle.

[0115] It should be noted that the sample bird's-eye view feature map can be determined based on the perception information. The sample bird's-eye view feature map can be rasterized perception information generated under the bird's-eye view (BEV) spatial representation. The sample bird's-eye view feature map can be directly obtained from the upstream perception module. The sample bird's-eye view feature map may include but is not limited to drivable area information, route information, and occupancy information.

[0116] It should be noted that the sample traffic participant driving information may include but is not limited to the location, speed, acceleration, direction, etc. of the traffic participant. The traffic participant may be other traffic participants in the driving environment of the vehicle. The traffic participants may include motor vehicles, non-motor vehicles, pedestrians, etc.

[0117] It should be noted that part or all of the execution subject of steps 201 to 205 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a model training platform on the network side, etc. This embodiment does not specifically limit this.

[0118] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0119] In this way, an end-to-end path planning model can be trained based on a sample bird's-eye view feature map corresponding to the sample ego-vehicle driving information, sample traffic participant driving information, and the ego-vehicle's perception information. This can achieve a more accurate and effective path planning model without using high-precision map data. While ensuring the performance and effectiveness of the path planning model, it also reduces the amount of data to be processed by the model and reduces the processing computing power overhead of the overall path planning model, thereby ensuring the reliability and performance of the path planning of the autonomous driving vehicle.

[0120] Optionally, in a possible implementation of this embodiment, the feature coding network may include a self-vehicle feature coding network, a bird's-eye view feature coding network, and a traffic participant feature coding network. In step 202, first, the self-vehicle driving feature coding information can be obtained based on the self-vehicle feature coding network and the sample self-vehicle driving information. Secondly, the bird's-eye view feature map coding information can be obtained based on the bird's-eye view feature coding network and the sample bird's-eye view feature map. Thirdly, the traffic participant driving feature coding information can be obtained based on the traffic participant feature coding network and the sample traffic participant driving information. Thirdly, the self-vehicle driving feature coding information, the bird's-eye view feature map coding information, and the traffic participant driving feature coding information are fused to obtain fused feature coding information.

[0121] In this implementation, the bird's-eye view feature encoding network may include a spatial feature encoder and a position feature encoder.

[0122] In a specific implementation process of this implementation, first, the spatial feature encoder can be used to perform spatial feature encoding processing on the sample bird's-eye view feature map to obtain the result of the spatial feature encoding processing. Secondly, the position feature encoder can be used to perform position feature encoding processing on the sample bird's-eye view feature map to obtain the result of the position feature encoding processing. Thirdly, the result of the spatial feature encoding processing and the result of the position feature encoding processing can be added to obtain the bird's-eye view feature map encoding information.

[0123] In this implementation, the spatial feature encoder may be an encoder based on a convolutional neural network (CNN). The spatial feature encoder may be used to process the spatial feature information in the bird's-eye view feature map.

[0124] In a specific implementation process of this implementation method, the sample bird's-eye view feature map can be input into a CNN-based encoder, and the result of the spatial feature encoding processing of the bird's-eye view feature map can be output.

[0125] Here, the spatial feature information in the sample bird's-eye view feature map may include edge information, shape information, and the like.

[0126] In this implementation, the position feature encoding may be an encoder based on a multi-layer perceptron. The position feature encoding may be used to process the position information in the bird's-eye view feature map.

[0127] In another specific implementation process of this implementation method, the sample bird's-eye view feature map can be input into the position feature coding based on the multi-layer perceptron, and the result of the position feature coding processing of the bird's-eye view feature map can be output.

[0128] Here, the location feature information in the bird's-eye view feature map may include coordinate information and the like.

[0129] In this implementation, the sample self-vehicle feature encoding network may include a self-vehicle driving feature encoder. The self-vehicle driving feature encoder may be an encoder based on a multi-layer perceptron. The self-vehicle driving feature encoder may be used to process sample self-vehicle driving information. The self-vehicle, i.e., the vehicle driving information may include the vehicle's position, speed, acceleration, direction, etc.

[0130] In this implementation, the sample traffic participant feature encoding network may include a traffic participant driving feature encoder. The traffic participant driving feature encoder may be an encoder based on a multi-layer perceptron. The traffic participant driving feature encoder may be used to process the driving information of the traffic participant. The driving information of the traffic participant may include the position, speed, acceleration, direction, etc. of the traffic participant.

[0131] In this implementation, the path planning model may further include a concatenation network, which may be a concatenation layer (Concat) network.

[0132] In another specific implementation process of this implementation method, the splicing network is used to fuse the vehicle driving feature coding information, the bird's-eye view feature coding information, and the traffic participant driving feature coding information to obtain fused feature coding information.

[0133] Optionally, in a possible implementation of this embodiment, the path planning model may include a connector. In step 203, the connector may be used to convert the fused feature encoding information to obtain the converted fused feature encoding information.

[0134] In this implementation, a connector can be used to convert the fused feature encoding information into information in the form of tokens to be input into the large language model network. The connector can be based on a connector of a multi-layer perceptron structure.

[0135] Optionally, in a possible implementation of this embodiment, the path planning model includes a decoding network. In step 204, first, the fused feature encoding information after the conversion process can be input into the large language model network to obtain the output result of the large language model network. Secondly, the output result of the large language model network is decoded using the decoding network. Thirdly, based on the result of the decoding process, the output result of the path planning model is obtained.

[0136] In this implementation, the decoder may be a decoder based on a transform structure. Preferably, the decoder may be used to decode the path information of the output result of the large language model network.

[0137] In this implementation, the large language model network may include any of the large language models with different parameter amounts. Exemplarily, the large language model network may include any of the models with 7 billion, 13B, 34B, and 70B parameter amounts. The large language model network may also include, but is not limited to, a large language model based on internlm, a large language model of llama, and a large language model of vicuna.

[0138] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the training method of the path planning model of this embodiment. For a detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0139] In order to better understand the method of the embodiment of the present application, the method of the embodiment of the present application is described below with reference to the accompanying drawings and specific application scenarios.

[0140] Figure 3 is a flow chart of a path planning method provided by another embodiment of the present application, such as Figure 3 In this embodiment, the path planning method may include:

[0141] Step 301: Acquire training sample data, where the training sample data includes sample ego-vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego-vehicle's perception information.

[0142] In this embodiment, the sample bird's-eye view feature map may include drivable area information (Drivable Area), route information (Route), and occupancy information (Occupancy).

[0143] In this embodiment, the sample ego information, namely, the sample ego information, may include the position, speed, acceleration, direction, etc. of the vehicle.

[0144] In this embodiment, the sample traffic participant driving information, that is, the sample traffic participant information (agentinformation) may include the location, speed, acceleration, direction, etc. of the traffic participant.

[0145] Step 302: input the training sample data into the path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data.

[0146] Step 303: convert the fused feature coding information to obtain the converted fused feature coding information.

[0147] Step 304: Based on the large language model network of the path planning model and the fused feature encoding information after conversion processing, an output result of the path planning model is obtained.

[0148] Step 305: Based on the training sample data and the output result of the path planning model, the path planning model to be trained is updated and trained to obtain a trained path planning model.

[0149] Step 306: Obtain a bird's-eye view feature map corresponding to the driving information of the vehicle to be processed, the driving information of the traffic participants, and the perception information of the vehicle.

[0150] Step 307: input the vehicle's driving information, traffic participant's driving information, and the bird's-eye view feature map into a pre-trained path planning model to obtain an output result of the path planning model.

[0151] In this embodiment, the bird's-eye view feature map may include drivable area information, route information, and occupancy information. All information in the bird's-eye view feature map may be generated in real time based on the collected perception information. Route information may be obtained from navigation information.

[0152] In this embodiment, the vehicle driving information, that is, the vehicle information, may include the vehicle position, speed, acceleration, direction, etc. of the current and historical frames of the path planning.

[0153] In this embodiment, the traffic participant driving information, that is, the traffic participant information may include the position, speed, acceleration, direction, etc. of the traffic participant in the current and historical frames of the path planning.

[0154] In this embodiment, the pre-trained path planning model may include a feature encoding network and a large language model network.

[0155] Step 308: Based on the output result of the path planning model, obtain the path planning information of the vehicle.

[0156] In this embodiment, the output result of the path planning model may include the position information, speed information, and orientation information of the vehicle for a planned period of time in the future.

[0157] Here, the planned trajectory information of the vehicle can be obtained based on the planned position information, speed information, and direction information of the vehicle in the future, and the planned trajectory information of the vehicle can be used as the path planning information of the vehicle.

[0158] In this embodiment, Figure 4 FIG. 1 is a schematic diagram of an application scenario of a path planning method provided by another embodiment of the present application. Figure 4 As shown, in this application scenario, the path planning model may include a feature encoding network, a splicing network, a connector, a large language model network, and a decoding grid. The feature encoding network may include a self-vehicle feature encoding network, a bird's-eye view feature encoding network, and a traffic participant feature encoding network. The bird's-eye view feature encoding network includes a spatial feature encoder and a position feature encoder. After the bird's-eye view feature map corresponding to the self-vehicle driving information to be processed, the traffic participant driving information, and the self-vehicle perception information are input into the path planning model, the path planning information of the self-vehicle is output. The path planning information of the self-vehicle includes the location information, speed information, and orientation information of the self-vehicle for a planned period of time in the future.

[0159] For example, in this embodiment, Figure 4As shown in the figure, at the input end of the path planning model, the high-precision map related information may not be input, but the rasterized perception information generated based on the bird's-eye view spatial representation may be input, that is, the bird's-eye view feature map of the perception information of the vehicle, which may include the drivable area information, navigation route information, and occupancy information. All information of the bird's-eye view feature map can be generated in real time through the perception information. At the same time, the input end also includes input information such as the vehicle driving information and the traffic participant driving information. Secondly, the bird's-eye view feature map corresponding to the vehicle driving information, the traffic participant driving information and the vehicle's perception information is respectively passed through the corresponding feature encoder (Encoder), that is, the vehicle driving information is input into the vehicle feature encoding network, the bird's-eye view feature map is input into the spatial feature encoder and the position feature encoder, and the traffic participant driving information is input into the traffic participant feature encoding network, and then the feature encoding information output by the three paths is spliced ​​to obtain the fused feature encoding information, and the fused feature encoding information is input into the large language model network in the form of token through the connector. The input feature information is used for feature learning and information aggregation inside the large language model through the attention mechanism and the fully connected layer, and finally the output result is processed in an autoregressive manner. Finally, the output token form of the large language model is further decoded by the decoding network to generate the final path planning information of the vehicle. The path planning information of the vehicle may include the position, speed, and direction of the vehicle in the future.

[0160] In this embodiment, Figure 5A It is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a scenario where a vehicle starts and bypasses a parked vehicle in front. Figure 5B It is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a vehicle turning scenario. Figure 5C FIG. 1 is a schematic diagram of the application effect of the path planning method provided by another embodiment of the present application in a scenario where a vehicle passes through a construction area. Figure 5A , Figure 5B , Figure 5C As shown in the figure, the purple area is the road surface, the white area and the gray area are the background, the red box is the motor vehicle among the traffic participants, the blue box is the pedestrian or bicycle, and the yellow box is the traffic cone. The trajectory composed of the red rectangle and the red point is the path of the human driver. The trajectory composed of the black rectangle and the blue point is the path obtained using the path planning model.

[0161] Here, it can be understood that the better the model's path matches the human driver's path, the better the processing performance and reliability of the path planning model. Figure 5A , Figure 5B , Figure 5CAs shown, in various driving scenarios, the reliability of the planning results of the path planning model in this method and the planning performance of the model are better.

[0162] In this way, by adopting the solution in this embodiment, after the end-to-end high-precision map-free autonomous driving path planning model based on the large language model is built, it can be trained using millions of sample data, and finally the entire path planning model can achieve excellent performance. The path planning model framework is simple and has a high performance ceiling. Moreover, as the number of parameters increases, the performance ceiling of the entire end-to-end large language path planning model can also be improved. Here, the complex traditional planning system is replaced by an end-to-end large language model, which has high development efficiency and is maintenance-friendly.

[0163] In addition, by adopting the solution in this embodiment, it is possible to achieve no dependence on high-precision maps. The entire path planning model eliminates the dependence on high-precision maps, reduces development, storage, and hardware costs, improves adaptability, and improves efficiency and safety.

[0164] In addition, by adopting the solution in this embodiment, the path planning model has strong generalization. The large language model has good generalization and can still achieve excellent results under zero-sample / few-sample training conditions. Compared with other non-large language model neural network systems, the large language model is not only data-driven, but can further integrate the knowledge stored in the network and be truly knowledge-driven like a human, thereby significantly improving the generalization ability of the system. At the same time, the large language model is easily compatible with multimodal input, making it easier to adapt to tasks related to autonomous driving.

[0165] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0166] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0167] Figure 6 A structural block diagram of a path planning device provided by an embodiment of the present application is shown as follows: Figure 6As shown. The path planning device 600 of this embodiment may include an acquisition unit 601, an encoding unit 602, a conversion unit 603, an output unit 604, and an acquisition unit 605. Among them, the acquisition unit 601 is used to obtain the self-vehicle driving information to be processed, the traffic participant driving information and the bird's-eye feature map corresponding to the perception information of the self-vehicle; the encoding unit 602 is used to input the self-vehicle driving information, the traffic participant driving information and the bird's-eye feature map into a pre-trained path planning model to obtain fused feature coding information based on the feature coding network of the path planning model and the self-vehicle driving information, the traffic participant driving information and the bird's-eye feature map; the conversion unit 603 is used to convert the fused feature coding information to obtain the converted fused feature coding information; the output unit 604 is used to obtain the output result of the path planning model based on the large language model network of the path planning model and the converted fused feature coding information; the acquisition unit 605 is used to obtain the path planning information of the self-vehicle based on the output result of the path planning model.

[0168] It should be noted that part or all of the training device of the path planning model of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a path planning platform on the network side, etc. This embodiment does not specifically limit this.

[0169] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0170] Optionally, in a possible implementation of this embodiment, the feature coding network includes a self-vehicle feature coding network, a bird's-eye view feature coding network, and a traffic participant feature coding network, and the coding unit 602 can be used to obtain the self-vehicle driving feature coding information based on the self-vehicle feature coding network and the self-vehicle driving information; obtain the bird's-eye view feature map coding information based on the bird's-eye view feature coding network and the bird's-eye view feature map; obtain the traffic participant driving feature coding information based on the traffic participant feature coding network and the traffic participant driving information; and fuse the self-vehicle driving feature coding information, the bird's-eye view feature map coding information, and the traffic participant driving feature coding information to obtain fused feature coding information.

[0171] Optionally, in a possible implementation of the present embodiment, the encoding unit 602 may be used to perform spatial feature encoding processing on the sample bird's-eye view feature map using the spatial feature encoder to obtain the result of the spatial feature encoding processing; perform position feature encoding processing on the sample bird's-eye view feature map using the position feature encoder to obtain the result of the position feature encoding processing; and perform addition processing on the result of the spatial feature encoding processing and the result of the position feature encoding processing to obtain the bird's-eye view feature map encoding information.

[0172] Optionally, in a possible implementation of this embodiment, the path planning model includes a connector, and the conversion unit 603 can be used to use the connector to convert the fused feature coding information to obtain the converted fused feature coding information.

[0173] Optionally, in a possible implementation of this embodiment, the path planning model includes a decoding network, and the output unit 604 can be used to input the fusion feature encoding information after the conversion process into the large language model network to obtain the output result of the large language model network; use the decoding network to decode the output result of the large language model network; based on the result of the decoding process, obtain the output result of the path planning model.

[0174] In this embodiment, the acquisition unit can acquire the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle, and then the encoding unit can input the driving information of the vehicle, the driving information of the traffic participants, and the bird's-eye view feature map into a pre-trained path planning model to obtain fused feature coding information based on the feature coding network of the path planning model and the driving information of the vehicle, the driving information of the traffic participants, and the bird's-eye view feature map. The conversion unit can convert the fused feature coding information to obtain the converted fused feature coding information, which can be output by the output unit based on the large language model network of the path planning model and the conversion processing. The processed fusion feature coding information is used to obtain the output result of the path planning model, so that the obtaining unit can obtain the path planning information of the vehicle based on the output result of the path planning model. Since the pre-trained path planning model can be directly used, the path planning information of the vehicle can be obtained only based on the driving information of the vehicle, the driving information of the traffic participants and the bird's-eye view feature map corresponding to the perception information of the vehicle. While ensuring the accuracy of the path planning result, the corresponding path planning result can be directly output based on the end-to-end planning model, avoiding the use of high-precision map related information, reducing computing power overhead, optimizing planning performance and efficiency, and thus improving the reliability of path planning of autonomous driving vehicles.

[0175] Figure 7 A structural block diagram of a training device for a path planning model provided by an embodiment of the present application is shown. Figure 7 The path planning model training device 700 of this embodiment may include a first acquisition unit 701 , a first obtaining unit 702 , a first conversion unit 703 , a second obtaining unit 704 and a first training unit 705 . The path planning model to be trained includes a feature encoding network and a large language model network, wherein a first acquisition unit 701 is used to acquire training sample data, wherein the training sample data includes sample vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the perception information of the vehicle; a first acquisition unit 702 is used to input the training sample data into the path planning model to be trained, so as to obtain fused feature encoding information based on the feature encoding network and the training sample data; a first conversion unit 703 is used to convert the fused feature encoding information to obtain the converted fused feature encoding information; a second acquisition unit 704 is used to obtain the output result of the path planning model based on the large language model network of the path planning model and the converted fused feature encoding information; a first training unit 705 is used to update the path planning model to be trained based on the training sample data and the output result of the path planning model, so as to obtain a trained path planning model.

[0176] It should be noted that part or all of the path planning device of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a model training platform on the network side, etc. This embodiment does not specifically limit this.

[0177] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0178] In this embodiment, the training sample data is acquired by the first acquisition unit, and the training sample data includes sample vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the perception information of the vehicle. Then, the first acquisition unit can input the training sample data into the path planning model to be trained to obtain fused feature coding information based on the feature coding network and the training sample data. The first conversion unit converts the fused feature coding information to obtain the converted fused feature coding information. The second acquisition unit obtains the output result of the path planning model based on the large language model network of the path planning model and the converted fused feature coding information. The first training unit can update and train the path planning model to be trained based on the training sample data and the output result of the path planning model to obtain a trained path planning model. Since an end-to-end path planning model can be trained based on a sample bird's-eye view feature map corresponding to sample ego-vehicle driving information, sample traffic participant driving information and ego-vehicle perception information, a more accurate and effective path planning model can be obtained without using high-precision map data. While ensuring the performance and effectiveness of the path planning model, it also reduces the amount of data to be processed by the model and reduces the processing computing power overhead of the overall path planning, thereby ensuring the reliability and performance of the path planning of the autonomous driving vehicle.

[0179] In the technical solution of this application, the user personal information involved, such as the collection, storage, use, processing, transmission, provision and disclosure of user images and attribute data, etc., complies with the provisions of relevant laws and regulations and does not violate public order and good morals.

[0180] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0181] According to an embodiment of the present application, further, an autonomous driving vehicle including the provided electronic device is provided, and the autonomous driving vehicle may include an L2 or higher level unmanned vehicle, for example, an unmanned family car, an unmanned commercial vehicle, an unmanned logistics vehicle, an autonomous driving logistics vehicle, etc.

[0182] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0183] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0184] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0185] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the method of path planning and the method of model training. For example, in some embodiments, the method of path planning and the method of model training may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method of path planning and the method of model training described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the path planning method and the model training method in any other appropriate manner (for example, by means of firmware).

[0186] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0188] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0190] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0191] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0192] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0193] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A path planning method, characterized in that: The method comprises: Acquire the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle; Inputting the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map into a pre-trained path planning model to obtain fused feature coding information based on a feature coding network of the path planning model and the self-vehicle driving information, the traffic participant driving information and the bird's-eye view feature map; Performing conversion processing on the fused feature coding information to obtain converted fused feature coding information; Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained; Based on the output result of the path planning model, the path planning information of the vehicle is obtained.

2. The method according to claim 1, characterized in that The feature coding network includes a vehicle feature coding network, a bird's-eye view feature coding network, and a traffic participant feature coding network. The feature coding network based on the path planning model and the vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map obtains fused feature coding information, including: Based on the vehicle feature coding network and the vehicle driving information, obtaining vehicle driving feature coding information; Based on the bird's-eye view feature coding network and the bird's-eye view feature map, obtaining bird's-eye view feature map coding information; Based on the traffic participant feature coding network and the traffic participant driving information, obtaining traffic participant driving feature coding information; The vehicle driving characteristic coding information, the bird's-eye view characteristic coding information, and the traffic participant driving characteristic coding information are fused to obtain fused characteristic coding information.

3. The method according to claim 2, characterized in that The bird's-eye view feature encoding network includes a spatial feature encoder and a position feature encoder, and obtaining bird's-eye view feature map encoding information based on the bird's-eye view feature encoding network and the bird's-eye view feature map includes: Using the spatial feature encoder, performing spatial feature encoding processing on the bird's-eye view feature image to obtain a result of the spatial feature encoding processing; Using the position feature encoder, performing position feature encoding processing on the bird's-eye view feature image to obtain a result of the position feature encoding processing; The result of the spatial feature encoding process and the result of the position feature encoding process are added together to obtain the bird's-eye view feature map encoding information.

4. The method according to claim 1, characterized in that: The path planning model includes a connector, and the conversion process is performed on the fused feature coding information to obtain the converted fused feature coding information, including: The connector is used to convert the fused feature coding information to obtain the converted fused feature coding information.

5. The method according to claim 1, characterized in that The path planning model includes a decoding network, a large language model network based on the path planning model and the fused feature encoding information after the conversion process, and obtaining an output result of the path planning model includes: Inputting the converted fusion feature encoding information into the large language model network to obtain an output result of the large language model network; Using the decoding network, decoding the output result of the large language model network; Based on the result of the decoding process, an output result of the path planning model is obtained.

6. A training method for a path planning model, characterized in that: The path planning model to be trained includes a feature encoding network and a large language model network, and the method includes: Acquire training sample data, the training sample data including sample ego-vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego-vehicle perception information; Inputting the training sample data into a path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data; Performing conversion processing on the fused feature coding information to obtain converted fused feature coding information; Based on the large language model network of the path planning model and the fused feature encoding information after the conversion process, an output result of the path planning model is obtained; Based on the training sample data and the output result of the path planning model, the path planning model to be trained is updated and trained to obtain a trained path planning model.

7. A path planning device, characterized in that: The device comprises: An acquisition unit, used to acquire the driving information of the vehicle to be processed, the driving information of the traffic participants, and the bird's-eye view feature map corresponding to the perception information of the vehicle; an encoding unit, configured to input the self-vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map into a pre-trained path planning model, so as to obtain fused feature encoding information based on a feature encoding network of the path planning model and the self-vehicle driving information, the traffic participant driving information, and the bird's-eye view feature map; A conversion unit, used for performing conversion processing on the fused feature coding information to obtain the fused feature coding information after conversion processing; An output unit, used for obtaining an output result of the path planning model based on the large language model network of the path planning model and the fused feature encoding information after the conversion process; The obtaining unit is used to obtain the path planning information of the vehicle based on the output result of the path planning model.

8. A training device for a path planning model, characterized in that: The path planning model to be trained includes a feature encoding network and a large language model network, and the device includes: A first acquisition unit is used to acquire training sample data, wherein the training sample data includes sample ego vehicle driving information, sample traffic participant driving information, and a sample bird's-eye view feature map corresponding to the ego vehicle's perception information; A first obtaining unit, configured to input the training sample data into a path planning model to be trained, so as to obtain fused feature coding information based on the feature coding network and the training sample data; A first conversion unit is used to convert the fused feature coding information to obtain the converted fused feature coding information; A second obtaining unit is used to obtain an output result of the path planning model based on the large language model network of the path planning model and the fused feature encoding information after the conversion process; The first training unit is used to update the path planning model to be trained based on the training sample data and the output result of the path planning model to obtain a trained path planning model.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

11. An autonomous driving vehicle comprising the electronic device as claimed in claim 9.

Images